# MirageOS community call ### 24/06/2024 https://www.timeanddate.com/worldclock/fixedtime.html?msg=MirageOS+call&iso=20240624T07&ah=1 Participants: Pierre, Thomas G., Samuel, Virgile, Hannes, Fabrice Agenda: - network API thoughts https://github.com/mirage/mirage/issues/1550 - 4.5.2 release https://github.com/mirage/mirage/issues/1547 (with questions about boot parameters) - ocaml-solo5 and OCaml 5 Notes: #### Network API questions - Hannes: how did the current API occur? was it guided by performance reasoning? - Thomas: What API would you like to have? - Hannes: Something that is more flexible where I don't need to instantiate functors manually (but can pass callbacks) - Thomas: Hard to find the right API due to efficiency. It was designed at the same tim as openvswitch appeared (https://github.com/mirage/ocaml-openflow). - Hannes: maybe something like BPF (as tcpdump uses for filters) would be nice: flexible and can compile to efficient code -- but there's no OCaml implementation that I can find - Thomas: did anyone look at the openflow implementation? - Thomas: a bit more dynamic would be nice (liking adding a new callback on events), but you've to design it properly to meet the performance goals) - Hannes: for applications (web server etc.), the current API is ok, but for lower layer things (DHCP, NAT, VPN), it is tedious - Thomas: there's https://github.com/ygrek/ocaml-bpf - Thomas: for vpnkit (https://github.com/moby/vpnkit/) we had to add something to be more dynamic (like adding routes) - Virgile: how hard would it be to implement an eBPF in OCaml - Hannes: can't be that hard, maybe a weekend -- in the end you need the parser for the rule language, and then a "compiler" that outputs sequences of `Cstruct.get` and `equal` instructions - Virgile: maybe the path forward is to go with a BPF implementation, play around, look at performance and flexibility #### 4.5.2 release - Hannes: still waiting for some opam packages to be merged - Hannes: the remaining question is when a mirage device can take some runtime argument, should that be defined in the mirage package (the alternative is that the keys are defined in each unikernel) -- e.g. not done for git-over-ssh - Thomas: maybe they can move to the ssh/git library? (Hannes: that'd be nice, then they could have proper types as well (and not only string and int)). #### OCaml 5 and ocaml-solo5 - Sam: we're still waiting for some more tests (esp. looking at memory profile of (long-running) unikernels) - Sam: There are open PRs for fixing mirage-www - Hannes: wants to test more, but no time before August. Would like to deploy on retreat.mirage.io and look at the memory usage; but had issue with monitoring. Really want to avoid memory usage increase. - Thomas: has merged the opam-monorepo PR for ppx_lwt - Pierre: uses the qubes-firewall with OCaml 5 since 2 weeks, works like before -- already got ~10% more bandwidth - Hannes: one thing could as well be to measure the mirage-www unikernel side-by-side OCaml 4 and OCaml 5 (2 unikernels on two IP addresses) - Hannes: I use mirage-monitoring to gather statistics, and then Grafana and Influx -- at the same time albatross is gathering statistics from the host system about the unikernel and reports these into the same Influx database - Thomas/Hannes: we want to have a Grafana for mirage.io on the mirage-www host. If anyone has time to set this up, please volunteer for doing that. - Virgile: I can do that. Hannes: Please send me your ssh public key, I'll set you up with an account. Virgile: sent it on Matrix #### Unikraft update - Sam: hope to have a first PoC for the OCaml-solo5 equivalent soon (with qemu) - Sam: question: is clang used for MirageOS compilations? - Hannes: I use clang since a long time, and MirageOS compiles (nowadays) fine with it. This doesn't mean that any unikraft & MirageOS must be working on FreeBSD/clang - Sam: unikraft seems to be Linux-specific - Sam: had a small example for quite some time, try to automate the things that need manual intervention (in respect to the build system etc.) - Sam: unikraft releases often, their most recent release doesn't work on my arm setup - Sam & Fabrice: starting to work on the Mirage tool - Thomas: a "HOWTO write a new MirageOS backend" (documentation) would be a great outcome - Sam: at configure time, you decide on the backend (qemu, firecracker, kvm, xen) #### next meeting in two weeks ### 27/05/2024 Participants: Pierre, Thomas G., Shakthi, Sabine, Hannes, Samuel Agenda: - mirage 4.5.0 and 4.5.1 released \o/ - the dune-variant approach for defunctorisation - OCaml-solo5 goes OCaml 5.2 - merge and add a branch for Ocaml 4.14? - next meeting june 10th 09:00 CEST Notes: #### mirage 4.5 - Hannes: we have made a new release of Mirage!! Mirage 4.5 splits runtime and configuration time keys. 4.5.1 fixes a few bugs (and adds one breaking change). mirage-skeleton has been updated - all Hannes' unikernels as well. Mirage doesn't depend on mirage-runtime anymore. Summary: seems to work great! - Pierre: qubes-mirage-firewall also uses mirage 4.5 - Thomas: 4.5 was released just before the retreat, so we had time to find some bugs in that #### dune variants / defunctorisation - Hannes: Lucas added dune variant in Dune when we released mirage4. The idea was to use this but we dropped the idea at the time. Investigated using dune variants instead of functors durung the retreat. The result: 4.5.1 now uses dune variant for mirage-bootvar (used to be 4 repositories! now just one that uses variants and works nicely). The code is much clearer: a single repo - easier to compare the various implementation for unix/xen/solo5. Bigger topic: could we generalise this to the entire network stack. Also for the time device. Hit a bug in Dune (https://github.com/ocaml/dune/issues/10460) - found a small repro case and reported to Dune. Rudi found a fix and Dune master is now fixed. How do we move forward now? - mirage-time defunctorisation https://github.com/mirage/mirage/pull/1529 - Thomas: where to get rid of functors? - everywhere? at the application side, everything should be static - but when you write libraries, it is a bit unclear - also for testing - if you look at the file system / block device stack, you may need some flexibility here - it is useful to get rid of functors where we can - Thomas: if there's a way to keep functors when we develop the libraries, but the exposed interface is very simple (no functor) - Thomas: error messages -- if you're using Time, but forget the dependency, you get not so nice error messages - Hannes: - Time/Clock/Random: no need for functor - mirage-net/ethernet/arp: no need for a functor - Block: CCM/partioning; VPN: also provide a network stack - here having a functor makes sense - Thomas: we expose some internal complexity (what is functorised / what is not?) to the user - how to communicate to the user whether a device is functorised or not (put it on the extra_deps vs in the type (`@->`) and application (`$`)) - Hannes: maybe `extra_deps` is not the right thing -- since we want to depend only on `mirage-time` / `mirage-clock` with the target-specific implementation - Samuel: we can add a dependency on mirage-time / mirage-clock in all the unikernels - Thomas: for the RNG it is a bit more complex, since we need to call the `initialize` function - Samuel: maybe we could always link time and clock with the unikernel - everyone: good idea! - Hannes: let's wait for the Dune release with the fix about dune variants - and make progress on time/clock! - Pierre: this is a good move, but the mirage-block-ccm should stay as is :D - Thomas: the question is where do we want this flexibility? Look at POSIX - Hannes: this is a great analogy - on UNIX systems we have tun/tap devices (for VPN etc.), and also block devices which support encryption -- so maybe at the device level it makes sense to have the abstraction #### OCaml 5 + solo5 - https://github.com/mirage/ocaml-solo5/pull/134 - Samuel: the behaviour with the GC changed a lot in OCaml 5, someone has a good example to test a long-running unikernel? - Hannes: maybe the retreat website, where we graph the memory usage (and thus have a comparison between last months and with OCaml 5) - Samuel: didn't manage to compile mirage-www yet, stuck in opam-monorepo - Thomas will look into that issue this week - Pierre: will swap his qubes-firewall to OCaml 5 this week, last week saw ~10-20% performance improvement with OCaml 5 - OCaml 5.2 supports compaction now, but we've to manually call `Gc.compact ()` - Pierre: at the early OCaml 5 stages, each domain allocated 2MB stack - we set the maximum domains to 1 in ocaml-solo5 - Samuel: when ocaml-solo5 is built, it sets the number of domains to 1 - Pierre: solo5 already reserves 1MB for the global stack #### MirageOS and Unikraft - Samuel: there are small examples, currently quite a lot needs to be done manually - Samuel: the question is who drives the build, unikraft wants to have control, mirage as well - Samuel: we will need something like ocaml-unikraft (similar to ocaml-solo5), at the moment `musl` is used - Samuel: there's some basic support for SMP in unikraft, but not yet enough for OCaml (scheduling) - Samuel: from performance perspective, one thing that is interesting is the support of ring interfaces - Thomas: the build system changes, we should talk with unikraft so that they're aware of what we do (and if they change stuff, we get informed) ### 06/05/2024 Participants: Thomas G, Kaushik, Rajesh, Pierre, Samuel, Vincent Balat, Hannes, Reynir, Pixie Agenda: - MirageOS retreat - Hannes: It was amazing! Good food! 17 participants (1/3 who came by bus/train). Lots of discussions about deployments. Lots of good discussion focused on documentation. ~10 PRs to the mirage website. Lots of different stuff happening. Blog posts are upcoming. Jules ported unikernels to mirage 4.5. Found a few issues and opened PRs. Thomas to check those. - Mirage and Dune virtual libraries (defunctorisation) - Hannes: found a few issues in dune variants. That seems to be a good idea. Discussed with Leandro and Vincent to port bigger libraires ([Riot](https://github.com/riot-ml/riot) and Ocsigen) to Mirage. It was not easy to them because of functors. Related to Time/Clock/Random issue with defunctorisation. Experimented with conditional compilation and dune variants. The complicated part is the network stack. But actually we do not need functors for the network stack. Current status: lots of overlays but succeeded to build the network example in the tutorial (using the TCP/IP stack) without functors. Still working on how to reproduce the Dune issues for the Dune team to fix. - The motivating example: think of a network device (mirage-net), for a given target (unix/solo5/xen), we'll only ever want the one network interface implementation (mirage-net-unix/mirage-net-solo5/mirage-net-xen) -- and don't want to mix them --> we can use dune variants (or conditional compilation) - another motivation is the clock implementation: who would want a unikernel with two different clocks - Thomas: really interesting to simplify the functor stack for the 80% of use-cases. What about virtual network: https://github.com/mirage/mirage-vnetif? What about the storage stack (where the "default storage" is harder to define) - mirage-kv - you may want to have a KV store backed by memory and a KV store backed by memory in the same unikernel -> functors are fine here - mirage-block - same argument as mirage-net -- we will only have a single implementation (depending on the target -- mirage-block-unix/mirage-block-xen/mirage-block-solo5) -- and not mix them - Thomas: interested in keeping the functors for "special" applications - ocsigen porting issues - (a) functors (you need to refactor your code to have a module that is functorized and the shared code elsewhere, so you need to call the code indirectly) - (b) static bundled files (maybe a tar archive) - (c) logging (there's syslog) - (d) configuration files (is dynlink supported by MirageOS? no. but there's a path to include the configuration into the binary) - (e) Makefile and build system (MirageOS builds in a dune monorepo) - OCaml 5 - Samuel: the PR for solo5 + OCaml 5.2 is opened. A few extra patches to be pushed soon. - Samuel: started exploring the unikraft backend with Fabrice (to be as similar as possible as the solo5 changes) - Pierre: to try on mirage-qubes-firewall; Samuel and Thomas will try to test it with mirage-www - Pierre: what's the status with io-page/allocations? Could you rebase the PR on the head of ocaml-solo5 to simplify testing? Samuel will - no changes to the allocator so should merge fine. - Ocsigen + Dream - kaushik: what is the difference / where does it sit? - kaushik: what does dream offer? is it a subset of ocsigen? - kaushik: ocsigen: fully end-to-end framework, dream: API for server - vincent: what you can do with dream, you can do with ocsigen - but ocsigen is a full stack, you can do more with it - vincent: dream is likely easier to use if you're only developing on the server side - vincent: with ocsigen, you can integrate client-side as well, and go the full way to mobile applications - thomas: dream works with mirageos, ocsigen not yet - thomas: mirage-www uses dream (it is only a static site) - vincent: people start with dream, and then want to add features and need to do everything manually -- think when you start where you want to go - next meeting in two weeks (may 20th) 09:00 CEST ### 29/04/2024 Participants: Thomas G, Shakthi, Magnus, Sam - Short meeting. Many people still traveling after the retreat, will meet again in one week. ### 08/04/2024 Participants: Magnus, Pierre, Kaushik, Vincent, Shakthi, Reynir, Hannes, Thomas, Samuel - Why are you here? - Kaushik: 2nd meeting, want to understand what he didn't know that he didn't know; use MirageOS in real-world problem solving - Vincent: working at Tarides on SpaceOS, still learning MirageOS, learn and understand what is going on; prepare the Marrakech retreat - Pierre: associate professor in network security, know and understand what people are working on in the MirageOS ecosystem, uses QubesOS (and the mirage-qubes-firewall) and MirageOS, is motivated to get rid of cstruct (based on bigarrays, there are issues (with q-m-f & bigarray due to malloc)), interested in the OCaml 5 release of MirageOS for better performance - Magnus: working on MirageOS since ~10 years, interested in the technology, worked on MirageOS in Docker, Tarides, now back to research -- working e.g. on IncludeOS/MirageOS performance with Alfred - Shakthi: works at Tarides on the ocaml.org team, fixing CI issues, looking at good first issues for newcomers - Samuel: engineer at Tarides, worked in the last months on OCaml 5 support for MirageOS (now solo5 works on OCaml 5.2 beta), works on Unikraft support for MirageOS, has interest in low-level stuff - Reynir: work in robur.coop with hannes and dinosaure, we develop MirageOS applications and maintain quite some libraries, interest to develop and deploy applications - Thomas: working on MirageOS for a while, happy to see the call starting again, still there are several parties interested in MirageOS, nice area of system research, interest: how to apply that to deployment - how to improve the IT landscape, MirageOS brings something new to IT (type-safe, memory-safe, formal verification): low-level systems with high-level languages, short term interest: CLI tool is more usable / friendly -- the OCaml ecosystem has evolved a lot since MirageOS started -- take ideas from MirageOS and upstream them to OCaml - Hannes: Interest in getting MirageOS used & deployed -- has had this goal for roughly a decade since he started on Mirage. Have done a PhD about formal verification of java programs. Would like to have more formally verified systems, so joined Mirage OS project where there is less mutable state. Worked on network protocols. Recent months worked on performance as we should have good performance when we run without kernel & user space. Not afraid of big API changes - they are a challenge. Big performance bottleneck is bigarray allocations, so recent PRs have been about replacing cstruct with string or bytes with promising benchmark results. How do we safely and pragmatically remove bounds checks? Worked on dependently typed systems such as Coq (now Rocq) and Idris where this is somewhat easier to express safely. Wants people to run their own services instead of using Google etc. Interested in getting reproducible builds for deployment reasons. Also interested in binary sizes. Getting more deployments requires getting more people involved => organizing retreats and hanging out on the weekly call to get more people interested/involved (since it is not feasible to do it alone). - Proposal for next MirageOS release 4.5.0 (Thomas) - Contents: what we have now in `main` (main update: runtime arguments) - When: before or after the retreat? - Proposal: Dune UX not quite ready for 5.0, better to release current version. Would be nice to release this before the retreat, so people there have an opportunity to test. So suggest tagging the release now and updating the documentation. Then continue discussion about other features. - Will make release before the retreat - Next: better Dune integration with Mirage - Design doc: https://gist.github.com/samoht/98127448fcf6a2483fa36099649fba5a -- feedback welcome! - PR: https://github.com/mirage/mirage/pull/1495 - Blockers: bugs in Dune - need experiments with the UX/UI (promotion?) - Next: OCaml5 + Mirage - PR almost ready: https://github.com/shym/ocaml-solo5/tree/ocaml-5.2 with its [Description](https://paste.sr.ht/~rustyne/0f76561709ca6c28852fae312e835e1059918f6e). - Hannes: this is very exciting. Just to be clear, the new ocaml-solo5 will only support OCaml 5.2, if we need releases for 4.x compatibility, we'll just branch off the latest release and put in changes - Thomas: it would also be nice if cross-compilation worked again on MacOS, can test - Next: `dune pkg` + Miragemail-betreuung bis nächsten montag übernehmen - The performance perspective: "no-cstruct" and bounds checking (Hannes) - opam overlay https://github.com/robur-coop/no-cstruct - Hannes: would be nice to have a mirage release that would support this as it needs changes to the mirage tool - Started thread to discuss using type system for bounds checking, but looks like this would require ppx right now, which I'd like to avoid: https://discuss.ocaml.org/t/bounds-checks-for-string-and-bytes-when-retrieving-or-setting-subparts-thereof/14422/6 - Interested in testing whether programs runs faster if bounds checks are removed from the compiler, how much faster is it? - `-unsafe` doesn't remove bounds checks from `String.get_*` - but a modified Stdlib (or compiler) emitting no bounds checks could do the trick - then measure on some real applications and take a look at the difference - Random/Clock/Time functors (Hannes) https://github.com/mirage/mirage/issues/1513 - As Hannes understands, a functor is useful if we use a single implementation multiple times (e.g. Set/Map -- a set of string, a set of int) - All Mirage unikernels are functorised over these. Have not seen different implementations of these, could we simplify the unikernels to not use functors for these? - we could replace that by the "linking trick" (dune variants) - testing sometimes (such as mirage-tcpip) uses a clock that runs faster than - would be great to cope with that - Next meeting in three weeks (Apr 29th 09:00 CEST), skip one week due to retreat -- https://www.timeanddate.com/worldclock/fixedtime.html?msg=MirageOS+call&iso=20240429T09&ah=1 ### 25/03/2024 Participants: Magnus, Pierre, Thomas, Vincent, Reynir, Samuel, Romain, Hannes, Shakthi - What do we want Mirage5 to look like? And when do we want to make a release? (Thomas) - runtime args - many changes in mirage tool, so should probably be in next release - Hannes: would be nice to release this early, could be part of ocaml5 - but could also release as 4.x - Ocaml 5 support - need ocaml5 support, at least single core support in solo5 - could release multicore support later - Hannes: ocaml5 doesn't need changes in all of mirage? thomas: we could separate it, but could also make mirageos5 the ocaml5 release. Multicore support will probably require more tooling changes (e.g. if we add support for new runtime), could start with single core support in solo5 - dune pkg? - support the new dune pkg system, alpha expected in may - Hannes: has already been delayed, so suggest not waiting for this. Thomas: should be ready for may this time - Romain: dune pkg aims to resolve some of the issues we have with opam monorepo, where can we discuss the issues with dune team? Thomas: team building and testing packages now, blocker at the moment is that packages are required to be relocatable. - https://github.com/ocaml/dune/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Package+Management+MVP%22 - Romain: to clarify the story about cross-compilation, only few packages requires it actually and mainly packages with C stubs (Zarith, digestif, mirage-crypto). So, we don't systematically need to "dunify" all packages (like dbuenzli packages). So it's good that `dune pkg` considers packages as a black box package which does not need dune to build - hannes: Good package to test x-compilation is Zarith/GMP - Romain: Zarith/GMP will still need an overlay however considering the status of the dunification of this package (https://github.com/ocaml/Zarith/pull/143) - No more Lwt? - larger discussion, may not be able to resolve this now, but start discussion - Hannes: this could be a separate discussion and will require some evaluation as well. Not sure we're ready to move away from this by next release. Several alternative schedulers - Thomas: Agree it's early, we need to experiment. Currently we see that Lwt binds are very slow (see irmin benchmarks, where they allocate a lot) so we would like to experiment with alternate solutions for Mirage - Thomas: ocaml5 support single core + runtime args could be an ocaml5 release, then later in the year we could add support for dune pkg and multicore support - Thomas: will write a suggestion for release candidate and we can discuss in two weeks - PRs: - [mirage/mirage#1495](https://github.com/mirage/mirage/pull/1495): refactor Dune files - If you have tooling issues (e.g. merlin not working properly) please let Thomas know and this could be fixed in this PR - Two conflicting design issues; if you only build a single unikernel in a repo then the workflow is ok, but doesn't match mirage skeleton - where we build 10 unikernels in one go after configuring each of them. Do we want the ability to have multiple unikernels in a single repo? Hannes: Never used this, usually build from each directory - Building MirageOS Library packages on Alpine 3.19 [Summary](https://hackmd.io/_0QAwb0rS72wDx52hkegPA). Reference: https://ocaml.ci.dev/github/mirage CI builds - Shakthi: trying to build for alpine in CI, have some patches. Are some other distro failures as well. I could start with a subset and fix common failures. - Thomas: seems to be working in regular CI, should investigate what the difference is. Nice to have Alpine support, so upstreaming would be good - MirageOS retreat in Marrakech (Hannes) - 18 people signed up! - Cstruct.t vs bytes/string - Hannes: working on removing cstruct in various repositories. E.g. mirage-crypto. Result is very nice, we see performance improvements. Plan is to remove cstruct in the whole network stack. Not planned in the tcp part of mirage-tcpip (as it's very big), but in the other packages in the stack. Also tools that use ocaml heap produce more interesting results, like statmemprof. - Thomas: what type of benchmarks? Hannes: micro benchmarks, performance measurements, throughput for symmetric ciphers. No improvement for large blocks, but for smaller there is (1024 and less) - mirage-flow 4.x and shutdown - Hannes: added shutdown function so you can close the connection, based on initial work from Dave. Have some PRs open, would be nice to have reviews. Thomas: Dave will take a look at mirage-tcpip - Romain: currently testing tls with shutdown. Works better with http - Next meeting will be April 8th, same time