-
Notifications
You must be signed in to change notification settings - Fork 9
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Documenter.jl
committed
Nov 3, 2023
1 parent
6ed011c
commit a5cfeb2
Showing
12 changed files
with
22 additions
and
19 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| {"documenter":{"julia_version":"1.9.3","generation_timestamp":"2023-11-03T12:49:11","documenter_version":"1.1.2"}} | ||
| {"documenter":{"julia_version":"1.9.3","generation_timestamp":"2023-11-03T15:23:54","documenter_version":"1.1.2"}} |
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| <!DOCTYPE html> | ||
| <html lang="en"><head><meta charset="UTF-8"/><meta name="viewport" content="width=device-width, initial-scale=1.0"/><title>Performance · HiddenMarkovModels.jl</title><meta name="title" content="Performance · HiddenMarkovModels.jl"/><meta property="og:title" content="Performance · HiddenMarkovModels.jl"/><meta property="twitter:title" content="Performance · HiddenMarkovModels.jl"/><meta name="description" content="Documentation for HiddenMarkovModels.jl."/><meta property="og:description" content="Documentation for HiddenMarkovModels.jl."/><meta property="twitter:description" content="Documentation for HiddenMarkovModels.jl."/><meta property="og:url" content="https://gdalle.github.io/HiddenMarkovModels.jl/alt_performance/"/><meta property="twitter:url" content="https://gdalle.github.io/HiddenMarkovModels.jl/alt_performance/"/><link rel="canonical" href="https://gdalle.github.io/HiddenMarkovModels.jl/alt_performance/"/><script data-outdated-warner src="../assets/warner.js"></script><link href="https://cdnjs.cloudflare.com/ajax/libs/lato-font/3.0.0/css/lato-font.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/juliamono/0.050/juliamono.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.2/css/fontawesome.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.2/css/solid.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.2/css/brands.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.16.8/katex.min.css" rel="stylesheet" type="text/css"/><script>documenterBaseURL=".."</script><script src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js" data-main="../assets/documenter.js"></script><script src="../search_index.js"></script><script src="../siteinfo.js"></script><script src="../../versions.js"></script><link class="docs-theme-link" rel="stylesheet" type="text/css" href="../assets/themes/documenter-dark.css" data-theme-name="documenter-dark" data-theme-primary-dark/><link class="docs-theme-link" rel="stylesheet" type="text/css" href="../assets/themes/documenter-light.css" data-theme-name="documenter-light" data-theme-primary/><script src="../assets/themeswap.js"></script></head><body><div id="documenter"><nav class="docs-sidebar"><a class="docs-logo" href="../"><img src="../assets/logo.png" alt="HiddenMarkovModels.jl logo"/></a><div class="docs-package-name"><span class="docs-autofit"><a href="../">HiddenMarkovModels.jl</a></span></div><button class="docs-search-query input is-rounded is-small is-clickable my-2 mx-auto py-1 px-2" id="documenter-search-query">Search docs (Ctrl + /)</button><ul class="docs-menu"><li><a class="tocitem" href="../">Home</a></li><li><span class="tocitem">Essentials</span><ul><li><a class="tocitem" href="../background/">Background</a></li><li><a class="tocitem" href="../api/">API reference</a></li></ul></li><li><span class="tocitem">Tutorials</span><ul><li><a class="tocitem" href="../tuto_builtin/">Built-in HMM</a></li><li><a class="tocitem" href="../tuto_custom/">Custom HMM</a></li><li><a class="tocitem" href="../debugging/">Debugging</a></li></ul></li><li><span class="tocitem">Alternatives</span><ul><li><a class="tocitem" href="../alt_features/">Features</a></li><li class="is-active"><a class="tocitem" href>Performance</a><ul class="internal"><li><a class="tocitem" href="#Reproducibility"><span>Reproducibility</span></a></li><li><a class="tocitem" href="#Remarks"><span>Remarks</span></a></li><li><a class="tocitem" href="#Numerical-results"><span>Numerical results</span></a></li></ul></li></ul></li><li><span class="tocitem">Advanced</span><ul><li><a class="tocitem" href="../formulas/">Formulas</a></li><li><a class="tocitem" href="../roadmap/">Roadmap</a></li></ul></li></ul><div class="docs-version-selector field has-addons"><div class="control"><span class="docs-label button is-static is-size-7">Version</span></div><div class="docs-selector control is-expanded"><div class="select is-fullwidth is-size-7"><select id="documenter-version-selector"></select></div></div></div></nav><div class="docs-main"><header class="docs-navbar"><a class="docs-sidebar-button docs-navbar-link fa-solid fa-bars is-hidden-desktop" id="documenter-sidebar-button" href="#"></a><nav class="breadcrumb"><ul class="is-hidden-mobile"><li><a class="is-disabled">Alternatives</a></li><li class="is-active"><a href>Performance</a></li></ul><ul class="is-hidden-tablet"><li class="is-active"><a href>Performance</a></li></ul></nav><div class="docs-right"><a class="docs-navbar-link" href="https://github.com/gdalle/HiddenMarkovModels.jl" title="View the repository on GitHub"><span class="docs-icon fa-brands"></span><span class="docs-label is-hidden-touch">GitHub</span></a><a class="docs-navbar-link" href="https://github.com/gdalle/HiddenMarkovModels.jl/blob/main/docs/src/alt_performance.md" title="Edit source on GitHub"><span class="docs-icon fa-solid"></span></a><a class="docs-settings-button docs-navbar-link fa-solid fa-gear" id="documenter-settings-button" href="#" title="Settings"></a><a class="docs-article-toggle-button fa-solid fa-chevron-up" id="documenter-article-toggle-button" href="javascript:;" title="Collapse all docstrings"></a></div></header><article class="content" id="documenter-page"><h1 id="Alternatives-performance"><a class="docs-heading-anchor" href="#Alternatives-performance">Alternatives - performance</a><a id="Alternatives-performance-1"></a><a class="docs-heading-anchor-permalink" href="#Alternatives-performance" title="Permalink"></a></h1><p>We compare performance among the following packages:</p><ul><li>HiddenMarkovModels.jl (abbreviated to HMMs.jl)</li><li><a href="https://github.com/maxmouchet/HMMBase.jl">HMMBase.jl</a></li><li><a href="https://github.com/hmmlearn/hmmlearn">hmmlearn</a></li><li><a href="https://github.com/jmschrei/pomegranate">pomegranate</a></li></ul><p>The test case is an HMM with diagonal multivariate normal observations.</p><ul><li><code>N</code>: number of states</li><li><code>D</code>: dimension of the observations</li><li><code>T</code>: trajectory length</li><li><code>K</code>: number of trajectories</li><li><code>I</code>: number of Baum-Welch iterations</li></ul><div class="admonition is-danger"><header class="admonition-header">Missing benchmarks?</header><div class="admonition-body"><p>The benchmarks are computationally expensive and we only run them once for each new release. If you don't see any plots below and the links are broken, you are probably on the dev documentation: go to the <a href="https://gdalle.github.io/HiddenMarkovModels.jl/stable/">stable documentation</a> instead.</p></div></div><h2 id="Reproducibility"><a class="docs-heading-anchor" href="#Reproducibility">Reproducibility</a><a id="Reproducibility-1"></a><a class="docs-heading-anchor-permalink" href="#Reproducibility" title="Permalink"></a></h2><p>These benchmarks were generated in the following environment: <a href="./assets/benchmark/results/setup.txt"><code>setup.txt</code></a>.</p><p>If you want to run them on your machine:</p><ol><li><p>Clone the <a href="https://github.com/gdalle/HiddenMarkovModels.jl">HiddenMarkovModels.jl</a> repository</p></li><li><p>Open a Julia REPL at the root</p></li><li><p>Run the following commands</p><pre><code class="language-julia hljs">include("benchmark/run_benchmarks.jl") | ||
| include("benchmark/process_benchmarks.jl")</code></pre></li></ol><h2 id="Remarks"><a class="docs-heading-anchor" href="#Remarks">Remarks</a><a id="Remarks-1"></a><a class="docs-heading-anchor-permalink" href="#Remarks" title="Permalink"></a></h2><h3 id="Julia-to-Python-overhead"><a class="docs-heading-anchor" href="#Julia-to-Python-overhead">Julia-to-Python overhead</a><a id="Julia-to-Python-overhead-1"></a><a class="docs-heading-anchor-permalink" href="#Julia-to-Python-overhead" title="Permalink"></a></h3><p>Since the Python packages are called from Julia with <a href="https://github.com/cjdoris/PythonCall.jl">PythonCall.jl</a>, we pay a small overhead that is hard to quantify. On the plots, we compensate it by subtracting the runtime of the same algorithm for the smallest instance (<code>N=1</code>, <code>D=1</code>, <code>T=2</code>, <code>K=1</code>, <code>I=1</code>) from all Python-generated curves.</p><h3 id="Allocations"><a class="docs-heading-anchor" href="#Allocations">Allocations</a><a id="Allocations-1"></a><a class="docs-heading-anchor-permalink" href="#Allocations" title="Permalink"></a></h3><p>A major bottleneck of performance in Julia is memory allocations. The benchmarks for HMMs.jl thus employ a custom implementation of diagonal multivariate normals, which is entirely allocation-free.</p><p>This partly explains the performance gap with HMMBase.jl as the dimension <code>D</code> grows beyond 1. Such a trick is also possible with HMMBase.jl, but more demanding since it requires subtyping <code>Distribution</code> from <a href="https://github.com/JuliaStats/Distributions.jl">Distributions.jl</a>, instead of just implementing <a href="https://github.com/JuliaMath/DensityInterface.jl">DensityInterface.jl</a>.</p><h3 id="Parallelism"><a class="docs-heading-anchor" href="#Parallelism">Parallelism</a><a id="Parallelism-1"></a><a class="docs-heading-anchor-permalink" href="#Parallelism" title="Permalink"></a></h3><p>The packages we include have different approaches to parallelism, which can bias the evaluation in complex ways:</p><table><tr><th style="text-align: right">Package</th><th style="text-align: right">States <code>N</code></th><th style="text-align: right">Observations <code>D</code></th><th style="text-align: right">Sequences <code>K</code></th></tr><tr><td style="text-align: right">HMMs.jl</td><td style="text-align: right">LinearAlgebra<sup class="footnote-reference"><a id="citeref-2" href="#footnote-2">[2]</a></sup></td><td style="text-align: right">depends<sup class="footnote-reference"><a id="citeref-2" href="#footnote-2">[2]</a></sup></td><td style="text-align: right">Threads<sup class="footnote-reference"><a id="citeref-1" href="#footnote-1">[1]</a></sup></td></tr><tr><td style="text-align: right">HMMBase.jl</td><td style="text-align: right">-</td><td style="text-align: right">depends<sup class="footnote-reference"><a id="citeref-2" href="#footnote-2">[2]</a></sup></td><td style="text-align: right">-</td></tr><tr><td style="text-align: right">hmmlearn</td><td style="text-align: right">NumPy<sup class="footnote-reference"><a id="citeref-2" href="#footnote-2">[2]</a></sup></td><td style="text-align: right">NumPy<sup class="footnote-reference"><a id="citeref-2" href="#footnote-2">[2]</a></sup></td><td style="text-align: right">NumPy<sup class="footnote-reference"><a id="citeref-2" href="#footnote-2">[2]</a></sup></td></tr><tr><td style="text-align: right">pomegranate</td><td style="text-align: right">PyTorch<sup class="footnote-reference"><a id="citeref-3" href="#footnote-3">[3]</a></sup></td><td style="text-align: right">PyTorch<sup class="footnote-reference"><a id="citeref-3" href="#footnote-3">[3]</a></sup></td><td style="text-align: right">PyTorch<sup class="footnote-reference"><a id="citeref-3" href="#footnote-3">[3]</a></sup></td></tr></table><p>We report each number of threads in <a href="./assets/benchmark/results/setup.txt"><code>setup.txt</code></a>. Since OpenBLAS threads have <a href="https://github.com/JuliaLang/julia/pull/50124">negative interactions</a> with Julia threads, we run the Julia benchmarks with a single OpenBLAS thread.</p><h2 id="Numerical-results"><a class="docs-heading-anchor" href="#Numerical-results">Numerical results</a><a id="Numerical-results-1"></a><a class="docs-heading-anchor-permalink" href="#Numerical-results" title="Permalink"></a></h2><h3 id="Low-dimension"><a class="docs-heading-anchor" href="#Low-dimension">Low dimension</a><a id="Low-dimension-1"></a><a class="docs-heading-anchor-permalink" href="#Low-dimension" title="Permalink"></a></h3><p>Full benchmark logs: <a href="./assets/benchmark/results/low_dim.csv"><code>low_dim.csv</code></a>.</p><p><img src="./assets/benchmark/plots/low_dim_logdensity_(D=1,T=1000,K=1).svg" alt/> <img src="./assets/benchmark/plots/low_dim_viterbi_(D=1,T=1000,K=1).svg" alt/> <img src="./assets/benchmark/plots/low_dim_forward_backward_(D=1,T=1000,K=1).svg" alt/> <img src="./assets/benchmark/plots/low_dim_baum_welch_(D=1,T=1000,K=1,I=10).svg" alt/></p><p><em>Here, pomegranate is not included because it is much slower on very small inputs.</em></p><h3 id="High-dimension"><a class="docs-heading-anchor" href="#High-dimension">High dimension</a><a id="High-dimension-1"></a><a class="docs-heading-anchor-permalink" href="#High-dimension" title="Permalink"></a></h3><p>Full benchmark logs: <a href="./assets/benchmark/results/high_dim.csv"><code>high_dim.csv</code></a>.</p><p><img src="./assets/benchmark/plots/high_dim_logdensity_(D=10,T=200,K=50).svg" alt/> <img src="./assets/benchmark/plots/high_dim_viterbi_(D=10,T=200,K=50).svg" alt/> <img src="./assets/benchmark/plots/high_dim_forward_backward_(D=10,T=200,K=50).svg" alt/> <img src="./assets/benchmark/plots/high_dim_baum_welch_(D=10,T=200,K=50,I=10).svg" alt/></p><p><em>Here, HMMBase.jl is not included because it does not support multiple sequences.</em></p><section class="footnotes is-size-7"><ul><li class="footnote" id="footnote-1"><a class="tag is-link" href="#citeref-1">1</a>affected by number of Julia threads</li><li class="footnote" id="footnote-2"><a class="tag is-link" href="#citeref-2">2</a>affected by number of OpenBLAS threads</li><li class="footnote" id="footnote-3"><a class="tag is-link" href="#citeref-3">3</a>affected by number of Pytorch threads</li></ul></section></article><nav class="docs-footer"><a class="docs-footer-prevpage" href="../alt_features/">« Features</a><a class="docs-footer-nextpage" href="../formulas/">Formulas »</a><div class="flexbox-break"></div><p class="footer-message">Powered by <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> and the <a href="https://julialang.org/">Julia Programming Language</a>.</p></nav></div><div class="modal" id="documenter-settings"><div class="modal-background"></div><div class="modal-card"><header class="modal-card-head"><p class="modal-card-title">Settings</p><button class="delete"></button></header><section class="modal-card-body"><p><label class="label">Theme</label><div class="select"><select id="documenter-themepicker"><option value="documenter-light">documenter-light</option><option value="documenter-dark">documenter-dark</option><option value="auto">Automatic (OS)</option></select></div></p><hr/><p>This document was generated with <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> version 1.1.2 on <span class="colophon-date" title="Friday 3 November 2023 15:23">Friday 3 November 2023</span>. Using Julia version 1.9.3.</p></section><footer class="modal-card-foot"></footer></div></div></div></body></html> |
Large diffs are not rendered by default.
Oops, something went wrong.
Oops, something went wrong.