v0.4.0

SLIM June update: Fitting, benchmark script, bugfixes

This release has breaking changes.
The most notable ones are that output format has once again changed into a parquet format, the introduction of cleaner fish treatment and that multiprocessing is now enabled by default.

New:

• New run command: slim. Can be used as slim run, slim gui, slim fit etc.
• New fitter script to fit on report data. The script uses ray tune and can choose across a number of fitting algorithms
• New benchmark script to run simulations, collect aggregated results and generate plots (closes #205 )
• New notebook that showcases how to run slim, CI data and specific payoff plot (closes #212 )
• New cleaner fish treatment (closes #26 )
• New artificial environments (closes #225 )
• Implemented "dummy" treatment policy that parses treatment actions from public records (closes #220 )
• Paving the road towards spark/ray datasets integration
• The CI now uses mamba rather than conda to speed up the GH actions' runtime.
• Halved the number of spawned ray actors by using ThreadedActors (closes #227 ) . On my laptop with 4 cores this resulted in an 11% speed-up.

Bugfixes:

• Ensure fallowing erases L3 population as well (closes #224 )
• force Python's GC to close zombie processes (closes #223 )
• Ensure num_fish=0 does not cause a crash (closes #221 )

Breaking changes:

• The simulation API has been changed again: Config.name now carries the simulation name, therefore sim_name has been removed from the parameter list
• The dumper will now automatically dump its output to a zstd-compressed parquet format. (closes #226 ). Furthermore, each run will also serialise its configuration in a separate file.
• The API for loading a simulation output has been simplified again due to the removed need to convert pickled data to pandas. Pyarrow can convert between Table and pd.DataFrame very quickly, making the pandas serialisation redundant.

PR:

• Fitting, new treatments, new serialisation, bugfixes by @erolm-a in #228

Full Changelog: v0.3.0...v0.4.0

v0.3.0: Multiprocessing and Numba-based optimisations

What's Changed

Replaced the GenoDistrib class to support multiple genes in a more efficient manner. The class is Numba-powered - although we currently recommend to keep the optimisations off until other classes get reconverted (see #209 ). Also simplified busy dam calculation by not using queues as they incur into slow downs.

Introduced farm-level multiprocessing with a promised 4-8x speed-up compared to the serialised version. The way artefact saving is performed has changed in that full dumping has been relegated to debugging only.

Towards stability

We believe that this project has become mature enough to be ready for the general public. While some parameters are still off, we believe in the correctness of the model we have written so far.

The change in PRs:

• Feature/new genotype by @erolm-a in #208
• Fix RTD CI, closes #210 by @erolm-a in #213
• Feature/multiprocessing by @erolm-a in #214

Full Changelog: v0.2.2...v0.3.0

v0.2.3.rc0: Reinforcement Learning

What has changed

This release introduces a new RL framework. Some of the API has changed to make Farms execute an action provided by a Policy, which can be decided in the Config. Additionally, multiple treatments can now overlap and the payoff/reward function will discourage severely infected salmon.

Before reaching a 0.3.0 we would like to support more treatment types and multiple genotype mechanisms, and polish the documentation.

We now have three hardcoded treatment policies: no treatment, binomial and mosaic. More treatments are coming soon

• Feature/gym by @erolm-a in #200

Full Changelog: v0.2.2...v0.2.3.rc0

v0.2.2

v0.2.2: new Beta reLouse

The changes are mainly these three:

• The plots can now include real-world data as a reference (if available). Data are extracted from the Salmon Scotland and Marine Scotland, even though better data sources have been found and in principle may be integrated.
  

• Map for interfarm travelling visualisation. Here is an example (Loch Fyne).
  

• Plotting non-continuous data (i.e. where the save_rate parameter is greater than 1) now works

Breaking changes

The configuration schemas have once again. The simulation API, however, has not (with the exception of GenoDistrib.allele_names being a dictionary rather than just a list).

Gory details

• Formatting by @erolm-a in #191
• Feature/report scraping by @erolm-a in #197

Full Changelog: v0.2.1...v0.2.2

Beta Relouse

This is a minor release

GUI changes:

• stroke lines are ticker, different stroke styles and lines should make them more readable
• aspect ratio was fixed (although in a hacky way)
• solved some UI glitches

Model changes

• made fish background and lice-induced mortality more sensible
• external pressure now takes a linear integration factor as input to further smooth the change once eggs are released.
• introduced dump compression via lz4 compression

The documentation has been updated to better describe the model workflow and fix a number of typos and improve the language.

What's Changed

• 174 improved plots by @erolm-a in #175
• 176 lzma by @erolm-a in #177
• Language changes model doc by @magicicada in #180
• 178 new exports by @erolm-a in #179
• Docs/improved docs by @erolm-a in #187

Full Changelog: v0.2.0...v0.2.1

Beta SLIM release: thermolicer, dynamic external pressure, restructuring and documentation

New

We support a new treatment type: thermolicer. We chose to implement it as it was easier to implement (compared to a dynamic population of cleaner fish) and had interesting side effects. Mortality data has been collected from Jensen 2017 and the survey script has been updated as usual.

The external pressure has been changed to be a linear function on the rolling average of released offsprings for the past T days (T being set to 30). The actual genotype distribution of the external pressure mutates and is defined by a Dirichlet-Multinomial process. In short, the more farms use a lake the more lice enter the reservoir and return to the cages regardless of the hydrodinamics of the loch.

We have also made the project easier to run and understand. In short, the project has been restructured so that to remove the need for refactored and simply put everything inside src, now renamed as slim to better adhere with standards. Thus the new command to run the project will be python -m slim.SeaLiceMgmt.

Lastly, we introduced documentation hosting via Sphinx+ReadTheDocs and updated class/modules docstrings accordingly. The process is not complete but a solid framework has been given.

PRs

• Improved mortalities, new thermolicer by @erolm-a in #164
• Feature/dynamic extpressure by @erolm-a in #165
• Sphinx #106 by @erolm-a in #171

Full Changelog: v0.1.0...v0.2.0

v0.1.0

First Beta reLouse: a star pandemic is born!

This release highlights on performance improvements and stability. The changes between this and the last version are somewhat massive both in terms of code restructuring, algorithmic implementation and outcome.

However, the main lifecycle of the program has not changed. The main changes are in how some key formulae work with regards to: mating, reproduction, evolution etc. and in general our visualisation tools have received a discrete overhaul.

Profiling and micro-optimisations

GenoDistrib, the key datatype in dealing with genotype distributions, has been reworked to be faster. The changes are in the order of microseconds, which may not seem like a lot but may sometimes constitute a bottleneck. A few other functions have been heavily optimised to be faster especially in hot paths.

External pressure

The total external pressure per farm is now proportional to the number of cages per farm. This is enough to trigger a pandemic.

Evolutions

We decided to drop Poisson distributions for age generation as they would encourage ages way lower than the threshold resulting in very few lice evolving.
The new formula is extracted from the Aldrin paper and internally emulates an age-distribution, then manually generates quantiles from the median development rate from a Weibull distribution and calculates average evolution rates from such event expectations. This is meant to be used in the following cases: L1->L2, L3->L4, L4->L5(m/f).

This resulted extremely effective especially as it would rise the chances of evolving between levels, but comes to a (slight) disadvantage of being somewhat slower. Therefore, sampling is performed no more than 1000 times to get a fairly reasonable expectation. In theory one could numerically compute thee expected development rate between stages (a constant) and apply a Gaussian over it, but for now this approach suffices.

Matings

The original spirit (Mendeleev-based matching) is preserved but the implementation has been changed to be O(1) (see #154 ). Similarly, a number of functions like dam/sire selection has been changed to be done in one go with a hypergeometric distribution. See #154

Furthermore, we spotted that currently mating code correctly calculates N_E eggs to extrude but actually generates O(N_E^2) eggs because each single "mating" will thus produce N_E eggs rather than being distributed.

The proposed solution fixes both problems at once.

This fix alone makes the simulation hundreds of times faster. For example, in a pandemic-like scenario with millions of eggs a single day update could take seconds while now the speed is consistently above 5 iterations per second (see below). Profiling data shows that after 80 days before this patch (runtime of ~1 minute) up to 22 seconds would be (cumulatively) spent on generate_eggs_discrete. Now this number dropped to 0.334 s with get_num_matings taking 0.4807 s overall, meaning the average time is 108 us /153 us .

Variance, treatment mortality etc.

All such functions have in common that they would sample on some quantity depending on lice population rather than being constant time. Due to a slip it was mentioned in the comments but not done ultimately.

Treatment mortality used to take overall 12.47s or ~4ms s per iteration, thus in a total of 6 it/s. Now it takes 3.12s, average runtime less than 1ms, thus a 4x speedup. The total runtime to emulate one year reduced from one minute to 36 seconds (~2x speedup).

Note that, when serialisation is disabled we get a speedup from 35 seconds to 15 seconds - as to be expected.

I believe we have collected evidence to close #145 .

Multiple testing environments

A new farm environment, test_Fyne, has been introduced. Users are now free to make experiments in Fyne without caring to break test cases. This has been possible for a while.

Visualiser

The visualiser can now plot fish population and mortality, treatment visualisation, offspring distribution, convolutions, the monte carlo walk from the optimiser and also provides an extra qtconsole in another tab.

Miscellanea

• Console outputs will not spam anymore as progress is reported via a tqdm
• Dropped support for quantitative mating
• Bugfixing on a number of functions that would break invariants
• Refactoring on the visualiser to be more modular

Changelog in PRs

• Profiling by @erolm-a in #141
• Extend visualiser features by @erolm-a in #149
• Feature/visualiser clues by @erolm-a in #151
• Evolutions, matings, infection variance, bugfixes by @erolm-a in #155

Full Changelog: v0.0.2...v0.0.3

second alphaLouse release

New

• Fish mortality also takes into account intoxication and increased stress levels due to treatments.

• Implemented a basic game theoretic framework: multiple farms act asindependent agents that can only communicate with a central organisation.

  Current workflow:

  1. Each farm decides on its own the rate at which to sample the cages.
  2. A farm will thus send this result back to the Org. (implementation note: farms are not directly instructed to do so, they merely generate sampling events and it is up to the Org to process them).
  3. If the sampled infection rate (Scotland:L5f/num_fish) > threshold (0.5) then send a treatment command to all the farms. The incriminated farm must treat unless no cage is eligible for treatment, while other farms may have a defection probability p_i depending of the i-th farm.
     Treatment eligibility depends on a few factors including whether another treatment is currently being administered, a cage isfallowing and so on).
     Furthermore, each day we calculate a payoff (roughly) equal to the fish population * mass at that day minus treatment cost, summed across farms.The objective is to maximise this function. Possible plan is to add other constraints (e.g. higher infection rate reduces the overall fish meat quality,hence reducing its price? environmental treatment costs?). Also we miss possible “social” effects of defecting.

• When fish die, adult males will have a chance to escape and find another host. So far this chance is independent from fish population, but we could not find enough data on this. When female lice die deaths are redistributed across free dams and “busy” dams, to avoid incurring into an infinite egg generator.

• Serialisation of the simulator for each day, or every few days (configurable via CLI) (currently not in master). Very useful for day-by-day visualisation but also to perform rollbacks while trying to debug invariants being broken.

• The CLI has changed considerably. Now only two parameter sare needed when running the model (python -m src.SeaLiceMgmt outputs/yourexperiment config_data/Fyne). Furthermore, automatic CLI parameters and help string are generated from params.json. Unfortunately the help guide is a bit finicky but we’re working on it. The big advantage is that generating multiple runs is as easy as making a for loopin bash.

• Sketched a UI (native, in PyQt) for the simulator (currently not in master).Right now it is only meant to be a visualiser, but it could also becom esomething more (e.g. a fully fledged GUI for the simulator). The rationale is that static charts provided by e.g. matplotlib are not interactable and not realtime (which would be nice to have), and also they may look less understandable to stakeholders.

Revamped

• Framework for multiple treatments. Other types of treament such asmechanical and thermal treatments (ie. “thermolicer”) are getting moreand more widespread - go-to option in Norwegian farms because of liceresistance. Some logic still needs to be moved.
• JSON encoder overhaul (although progressing towards deprecation, as pickle is the preferred format now)
• Plentiful of testing on edge cases
• Code cleanup in plenty of occasions and addition of a few more types.
• Genotype distributions are now properly wrapped in a Counter-like class: GenoDistrib. Compared to what was originally written, they are not truncated. The reason being, having negative frequencies makes it easier to write negative deltas. Obviously, this also means a few more checks have to be done every now and then to ensure non-negativity at the end of the day (pun intended).
• The driver code has been modularised so that the “model” is actually living in a standalone class. The idea is to also support Jupyter notebooks.

Deprecated

• Quantitative mechanism is still supported but largely untested.
• The protection window mechanism has been deprecated.

alpha reLouse

This model should perform all the basic functions involving the sea lice lifecycle.

Not ready for production use, and needs thorough testing/rewriting in many parts.