15 views
# MirageOS meetings every other Monday morning 10:00 - 12:00 CEST in https://meet.jit.si/MirageOS ## Topics - OCaml 5 performance - Unikraft - experiments with Bytes Cstructs and BA io-pages ## Meeting September 1st 10:00 - 12:00 CEST ## Meeting June 30th 10:00 - 12:00 CEST - Participants: Hannes, Pierre ### bytes vs bigarray - Pierre is disappointed by the performance on xen (see https://github.com/mirage/io-page/pull/72#issuecomment-2984504101) - Hannes we should test on spt/hvt, and also dig into the network implementations (to figure out where data is copied and how to avoid copies) - Hannes it is great, since that means that we'll finally have tooling from OCaml readily available (statmemprof, GC numbers, etc.) - Hannes we should instrumend and trace with https://github.com/robur-coop/memtrace-mirage ; as well as with perf (which works with kvm at least) - The plan is to test with https://github.com/palainp/simple-fw on solo5 targets, measure, re-iterate, and improve code ## Meeting June 16th 10:00 - 12:00 CEST - Participants: Pierre, Reynir, Hannes, Sam ### IO-page without Cstruct, and Cstruct.t without Bigarray - Pierre: have a branch with no bigarray in mirage-tcpip (see https://github.com/hannesm/mirage-tcpip/tree/no-bigarray) ``` cstruct.6.2.0 git git+https://github.com/hannesm/ocaml-cstruct.git#no-bigarray cstruct-lwt.6.2.0 git git+https://github.com/hannesm/ocaml-cstruct.git#no-bigarray io-page.3.0.0 git git+file:///home/user/mirage/io-page#no-cstruct mirage-net-xen.2.1.5 git git+file:///home/user/mirage/mirage-net-xen#io-page-ba mirage-xen.9.0.0 git git+file:///home/user/mirage/mirage-xen#new-iopage shared-memory-ring.3.2.1 git git+file:///home/user/mirage/shared-memory-ring#new-iopage shared-memory-ring-lwt.3.2.1 git git+file:///home/user/mirage/shared-memory-ring#new-iopage tcpip.9.0.1 (uninstalled) git git+https://github.com/hannesm/mirage-tcpip.git#no-bigarray vchan.6.0.2 git git+file:///home/user/mirage/ocaml-vchan#update-iopage-api vchan-xen.6.0.2 git git+file:///home/user/mirage/ocaml-vchan#update-iopage-api xen-gnt.4.0.2 (uninstalled) git git+file:///home/user/mirage/ocaml-gnt#new-iopage ``` ``` [2025-06-16 10:00:52] | ___| [2025-06-16 10:00:52] __| _ \ | _ \ __ \ [2025-06-16 10:00:52] \__ \ ( | | ( | ) | [2025-06-16 10:00:52] ____/\___/ _|\___/____/ [2025-06-16 10:00:52] Solo5: Bindings version v0.9.1 [2025-06-16 10:00:52] Solo5: Memory map: 32 MB addressable: [2025-06-16 10:00:52] Solo5: reserved @ (0x0 - 0xfffff) [2025-06-16 10:00:52] Solo5: text @ (0x100000 - 0x498fff) [2025-06-16 10:00:52] Solo5: rodata @ (0x499000 - 0x55afff) [2025-06-16 10:00:52] Solo5: data @ (0x55b000 - 0x8e4fff) [2025-06-16 10:00:52] Solo5: heap >= 0x8e5000 < stack < 0x2000000 [2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [net-xen frontend] connect 0 [2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [qubes.db] connecting to server... [2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [qubes.db] connected [2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [net-xen frontend] create: id=0 domid=1 [2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [net-xen frontend] sg:true gso_tcpv4:true rx_copy:true rx_flip:false smart_poll:false [2025-06-16 10:00:52] 2025-06-16T08:00:52-00:00: [INFO] [net-xen frontend] MAC: 00:16:3e:5e:6c:00 [2025-06-16 10:00:53] 2025-06-16T08:00:53-00:00: [INFO] [ethernet] Connected Ethernet interface 00:16:3e:5e:6c:00 [2025-06-16 10:00:53] 2025-06-16T08:00:53-00:00: [INFO] [ipv6] IP6: Starting [2025-06-16 10:00:53] 2025-06-16T08:00:53-00:00: [INFO] [ARP] Sending gratuitous ARP for 10.137.0.27 (00:16:3e:5e:6c:00) [2025-06-16 10:00:53] Fatal error: exception Invalid_argument("String.blit / Bytes.blit_string") [2025-06-16 10:00:53] Raised at Stdlib.invalid_arg in file "stdlib.ml", line 30, characters 20-45 [2025-06-16 10:00:53] Called from Ethernet__Ethernet_packet.Marshal.unsafe_fill in file "duniverse/ethernet/src/ethernet_packet.ml", line 63, characters 4-67 [2025-06-16 10:00:53] Called from Stdlib__Result.map in file "result.ml", line 25, characters 32-37 [2025-06-16 10:00:53] Called from Ethernet.Make.write.fill in file "duniverse/ethernet/src/ethernet.ml", line 107, characters 14-60 [2025-06-16 10:00:53] Called from Frontend.Make.write_already_locked.(fun) in file "duniverse/mirage-net-xen/lib/frontend.ml", line 304, characters 18-60 [2025-06-16 10:00:53] Called from Lwt.Sequential_composition.try_bind in file "duniverse/lwt/src/core/lwt.ml", line 2139, characters 10-14 [2025-06-16 10:00:53] Re-raised at Lwt.Miscellaneous.poll in file "duniverse/lwt/src/core/lwt.ml", line 3123, characters 20-29 [2025-06-16 10:00:53] Called from Xen_os__Main.run.aux in file "duniverse/mirage-xen/lib/main.ml", line 37, characters 10-20 [2025-06-16 10:00:53] Called from Dune__exe__Main.run in file "mirage/main.ml", line 4, characters 12-29 [2025-06-16 10:00:53] Called from Dune__exe__Main in file "mirage/main.ml", line 436, characters 2-7 [2025-06-16 10:00:53] Solo5: solo5_exit(2) called ``` - The issue is likely in mirage-net-xen passing a (too) small buffer to Ethernet, since the source location is a blit of the destination mac address (at offset 0 into the buffer) - Pierre: need to also investigate performance, since with the Cstruct using bytes does more copying - Hannes: I'm really happy this is moving forward. - Hannes: the ethernet implementation calls `Netif.write`, which calls the (mirage-net-xen) Frontend.write with the `~size` parameter, which then calls `write_no_retry`, which calls `write_already_locked` (`Cstruct.sub`) -- and I guess we should printf the `~size` parameter for further debugging - Hannes: in (mirage-net-xen) `write_already_locked` we call `fillf (Cstruct.sub cs_shared_block 0 size)` - Hannes: in (ethernet): `let size = eth_hdr_size + size` - Hannes: I'd recommend to printf the `size` - Pierre: there's some too small buffer passed somewhere - Hannes: I'm really happy to see this moving forward. The blocker in Mirleft was clearly the hanging of qubesdb/xen vchan. I'd love to think further about performance, and it seems the roadblock is lifted now :) ### Unikraft - Sam: The ocaml-unikraft package family has been released to the opam-repository - Virgile told that opam-monorepo is complaining with how the ocaml-unikraft packages are setup - Sam: we have patches for the mirage tool, also for net, block, time, etc. - we try to fix the small issues that are left, and it will be ready soon! - Hannes: did you do any further benchmarking? - Sam: not yet, there was a small bug: there are some extra bytes in each packet - unclear where they are coming from. we can safely just ignore them, they are always null bytes. - Sam: we plan to run again some benchmarks this week, and hope is that they'll be reasonably good - Pierre: in the end, it will be a potential replacement for solo5? - Sam: exactly, it is an alternative backend. for unikraft, you don't need solo5 tenders - you can run on qemu and firecracker. they have experimental ports for other backends they are working on. - Pierre: would it be possible to use with xen and qubes, or not? - Sam: unikraft has a backend for xen, but we didn't add it to the other layers - since I don't have a path to test it - Sam: I didn't write the code to generate the xen backend (in ocaml-unikraft) - Pierre: would be interesting, esp. if it delivers better performance - Hannes: which network stack do you use? - Sam: we initially used the unikraft one (which uses lwip), but now we use the OCaml one - Hannes: any news on multiple CPUs in unikraft? - Sam: not yet. it is on their roadmap. it is not in any released version. in some parts of the code I needed to debug, it is clear that it is monocore - Hannes: what is unikraft currently working on? do they have users? - Sam: they created a company to smoothen the deployment experience (unikraft cloud) - Hannes: I'm really happy you're pushing this work forward, so we can piggyback on the unikraft cloud, and at the same time we can evaluate the performance issues from their website (where solo5 is really slow), and figure out whether the performance is lost in solo5 or OCaml. ### OCaml compiler and cross-compiling - Sam: The OCaml 5.5 will include the patches needed for cross-compiling, so we don't need any further patches \o/ (apart from setting the number of domains to 1) - Hannes: For OCaml 5.4, do we have a patchset? - Sam: there's only a single PR left (to enable freestanding target), will open a PR on ocaml-solo5 and ocaml-unikraft when 5.4 is released - Hannes: from the OCaml release schedule, 5.4 is expected this summer (August) ### OxCaml - JaneStreet published their OxCaml project (including stack allocation, etc.) at https://oxcaml.org - MirageOS could use Oxcaml - it includes as well flambda2 - It would be interesting / nice to test the various compilers: - OCaml, OCaml with flambda1, OxCaml (with plain opam packages), OxCaml with annotated packages (to use e.g. stack allocation) - Sam: it is unclear how to use OxCaml with MirageOS, they seem to be based on the 5.2.0 compiler - and there would be various cross-compiling PRs needed to use OxCaml inside of MirageOS - Sam: unclear whether it is worth to backpatch OxCaml, or wait until they move to OCaml 5.3.0 or even 5.4.0. Maybe we'll find someone who has an idea of their roadmap. - Sam: it shouldn't be too hard to get something working, but it would be more work to upstream clean patches as merged. ## Meeting June 2nd 10:00 - 12:00 CEST - Participants: Pierre, Hannes, Virgile ### OCaml 5 performance - Virgile did some experiments in last 2024 about mirage-www and OCaml 5 - Notes about results (using OCaml 5.2): https://hackmd.io/@Firobe/ByKlnueNke - TL;DR: If set the space-overhead GC parameter to 60 (or 80), we get similar memory behaviour than OCaml 4 - "with OCaml 5, the GC parameters need to be tweaked" - with default space_overhead, there's ~2x CPU used; with space_overhead set to 60 (memory is 0.94x), 1.62x CPU is used - Pierre didn't measure the memory consumption with OCaml 5, the unikernel has 32MB assigned. The measure was throughput, and OCaml 5 was slightly better. - Virgile: from memories, the GC pacing is very bad under stress -- memory usage is exploding - Hannes: The compaction needs to be called manually (with OCaml5), so we need to figure places where to call it. - conversation about this on the Ocamllabs slack: https://hackmd.io/@Firobe/H19exkozxl - Pierre: Is compaction called from full_major? Virgile: No, it is not. - Pierre: when running a dns-resolver locally, there's quickly (after 10 - 20 minutes) an out of memory. This is due to a lot of Bigarray allocation. This is not only on OCaml 5. - Hannes: removing BAs is very desirable in general, but haven't been done yet mostly due to Xen which needs memory alignment - Hannes: availability of memory profiling tools will help figuring out the exact sources of the allocations and where efforts should be directed - Pierre: Didn't work too much on the io-page stuff. There's some bigarray usage in TCP/IP (checksum computation). Apart from that, it seems to be doable to get the io-page PR https://github.com/mirage/io-page/pull/72 merged. The cstruct-using-bytes branch works pretty nicely. - Hannes: about checksum computations, we can take the code in https://github.com/robur-coop/utcp/blob/main/src/checksum.ml which works for both bytes/string and Cstruct.t - Hannes will take a look and create a branch/PR for mirage-tcpip (under the assumption that Cstruct.t is backed by bytes) ### Unikraft - Virgile mentioned that this is mostly ready, there's a mail draft to be sent to the MirageOS mailing list - last bug to fix in network stack ### Security vulnerabilities - Hannes: will findings from hacksat challenge be reported to security@mirage.io - Virgile: Yes, all that are related to MirageOS, other open source findings will as well be upstreamed. - Hannes: when will you do that? At the time of when it is reported, or at the time the challenge ends? - Virgile: the current plan is to do that at the end. - Virgile: what is tested is one deployment of a HTTP/AF server. - Hannes: let's treat the 90 days period as a hard deadline to put out patches and reports ## Meeting April 28th 10:00 - 12:00 CEST - Participants: Fabrice, Pierre, Reynir, Hannes, Sam, Romain ### Defunctorization work Hannes worked on defunctorizing Mirage. It seems to work well and no complaints. Hannes: should we defunctorize the network stack next and the block devices? Pierre: is very happy with the current state of defunctorization. For the network stack, at least for Xen it will be tricky since we need two network interfaces -- for backend and frontend. Romain: Did an experiment about MirageOS and miou-solo5, and has a TCP stack without functors -- still an experimental project Reynir: If I understand, it goes a bit further with no functor at all Romain: https://git.robur.coop/dinosaure/experiment-miou-solo5 Romain: and a little article here: https://blog.robur.coop/articles/utcp_and_effects.html ### Unikraft Sam: the work on unikraft is close to be published Sam: performance for network is pretty good, for the block device the situation is less clear Fabrice: block solo5 outperforms the unikraft Hannes: from their website (unikraft), they claim much higher performance than solo5 Fabrice: network is fine, it is a little faster Sam: one of the next steps is to have a real benchmark with network Romain: do you know the performance issues with block devices and unikraft? Fabrice: it uses virtio device, quite different from the network one. performs badly if you do small sector (one sector at one time) operations -- much better if you operate on multiple sectors Fabrice: you can as well have multiple operations in flight with unikraft - which helps to shadow the single sector bad performance Romain: did some performance benchmarks in terms of mirage-tcpip and utcp, and there's a large gap in the benchmarks (using iperf3, we're near 1Gb/s with mirage-tcpip -- and with utcp 900 Mb/s) Fabrice: our network test served one file Romain: we have a really old unikernel which is compatible with iperf2, but the experiment above has iperf3 support Romain: mirage-tcpip is 1Gbits/s and utcp is 900 Gbits/s on our experimentation. The main question is about scheduling now (where miou differs from what lwt does) Sam: for getting it released, we populate the different repositories under the mirage organization, and open PRs for the repositories where we have unikraft patches ### utcp https://github.com/robur-coop/utcp Hannes: TCP/IP stack based on a formal model (HOL4, SML; manually translated SML to OCaml - Recently we found a mistranslation caused by a missing set of parenthesis). Hannes: mirage-tcpip: it works very well, but as discussed on the mailing list it has obscure semantics in certain cases. It is deeply in the LWT monad and has memory leaks. Hannes: Utcp has a pure, functional core with unit tests. Recently worked on performance with Romain. µtcp still lacks congestion control and newer features of TCP such as selective acknowledgement (which mirage-tcpip also doesn't implement). µtcp started off several years ago. We (in Robur) run it in production machines, and we have mostly worked on correctness and resource usage, and we still have correctness issues (failed assertions) and resource usage issues. Performance wise µtcp tries to stick to congression control and window sizes while mirage-tcpip doesn't try to adhere to a specific congression control algorithm or bound the memory usage. The gained interest of µtcp is also due to it not being tied to lwt and thus allows for other schedulers. Hannes: utcp is meant to replace only mirage-tcpip's src/tcp (ocamlfind tcpip.tcp) Romain: for the miou TCP/IP stack, we worked on a new IP stack (which is different from mirage-tcpip's one) ### Mirage CI Hannes: OCaml 5.4 support in ocaml-solo5 (ocaml-unikraft)? -- two different repositories but shared patches, also 5.4 has most patches upstream \o/ -- https://github.com/ocaml/ocaml/pull/13810 (maybe we can ask Antonin, Gabriel, Florian at the retreat whether that can make it to 5.4) Hannes: OCaml 5.3 is not yet tested in the Mirage CI Hannes: there's a PR from Tim about fixing the OCaml 5.2.1 support https://github.com/ocurrent/mirage-ci/pull/51 ### Remove bigarray from Cstruct Pierre: experimented with branches from Hannes that use cstruct where the buffer is Bytes.t Pierre: updated io-page to not rely on cstruct, but use bigarray directly Pierre: this currently works with QubesOS, a hello world runs nicely Pierre: ran into issues when running the network stack, mirage-tcpip is tightly coupled with cstruct and relies on the fact that cstruct is based on bigarray (esp. C stubs) Pierre: may use utcp to check whether that'll be good enough / work Pierre: need a careful review of the io-page API, since all its dependencies need updates Hannes: the only C code is the checksum code, no? in utcp we have pure OCaml checksum code, and we could use that in mirage-tcpip (we use some trick about bigarray to outperform the computation) ### Ownership of buffers on the IP level Romain: another experiment with the IP stack, with mirage-tcpip you have cstruct everywhere -- difficult to change to something else. with cstruct you want to not copy when you have a fragmented packet -- with bigarray and cstruct you can have a subview (without copying). the idea is to have a bigarray directly when you have a defragmented packet, and if it is fragmented you get a copy of the bigarrays Romain: with mirage-tcpip you have the question about ownership and fragmented/defragmented: do you have the ownership or not? Romain: My intuition is at the IP level, we should have a variant between a bigarray (defragmented) and a string (fragmented) -- if you have a bigarray you should care about the ownership Hannes: in practise, 99.9999% of IP packets are not fragmented ### Checksum code - performance investigations Hannes: maybe we should measure the utcp checksum code (OCaml code) and mirage-tcpip checksum code (C code) Romain: it is tricky due to memory layout, and also you've to take care that OCaml 5 C-FFI is different (and introduces a memory barrier), so check what your environment is before doing the benchmark (CPU cache) Hannes: maybe the checksum code could then be in a separate, independent package used by both utcp and mirage-tcpip Hannes: question about the performance focus: arm? x86? 64 bits only? also 32 bits? ### tcpip handling of RST Pierre: curious whether there was more communication about uTCP Hannes: there wasn't Pierre: I'll try reach out to them, one of the advisors is in my reasearch group ## ~~Meeting February 24th 10:00 - 12:00 CET~~ Cancelled ## Meeting February 10th 10:00 - 12:00 CET - Participants: Reynir, Pierre, Hannes, Sam, Pixie #### Blog article from Tarides about MirageOS and OCaml 5 - https://tarides.com/blog/2025-02-06-mirageos-on-ocaml-5/ ### GC compactor / free? - Sam expressed last time whether the malloc/free implementation was doing the right thing (esp. with OCaml 5 runtime, mmap/munmap); the compactor is punching holes in the memory - Sam: It is correct \o/ -- we don't expect any abort(), we may end up not really being able to compact the unikernel memory - Hannes: in Jane Street's ocaml-flambda, they have a PR for automatic running the compaction https://github.com/ocaml-flambda/flambda-backend/pull/3500 - Sam: it is not clear whether this is suitable for MirageOS unikernels ### Sam: for mirage-www the only thing we needed to tweak is the overhead parameter to have a similar memory profile than the OCaml 4 runtime - Sam: for some time, both OCaml 4 and OCaml 5 versions of the website were run in parallel, pushing every other connection to each unikernel. Unclear about the current state. - Hannes: I've not seen any graphs, and don't know about the current state - Sam: will propose to Virgile to publish his measures and say what the result is/was ### OCaml 5.3 support for ocaml-solo5? - Sam prepared a PR https://github.com/mirage/ocaml-solo5/pull/148 - Last Thursday the big PR was merged into OCaml@trunk https://github.com/ocaml/ocaml/pull/13735 - Now that should work for MirageOS - Sam quickly backported that PR to ocaml-solo5, but there's some CI issue - will work on it - Sam: there are only some minor patches needed in the future (5.4) for building the cross-compiler for freestanding/solo5 - Pierre did not yet try it, has a paper deadline ### A guide to enable the OpenBSD NetVM and then use QubesOS-MirageOS-firewall - https://forum.qubes-os.org/t/fortifying-sys-net-a-shift-to-openbsd/31973 ### Problem on building on OpenBSD - Sam: building solo5 is broken on OpenBSD? - Pierre: it was working two years ago when working on thread-local storage - https://github.com/Solo5/solo5/pull/573 - a first draft for OpenBSD 7.5 - Hannes: if we don't have any active OpenBSD users, it is not really worth the effort to port it. If there's some user, we should try to help them to get it compiling again :) ### Functors in MirageOS - Hannes MCLOCK, PCLOCK, TIME, RANDOM on their way out https://github.com/mirage/mirage/pull/1599 - Hannes replaces it with the "linking trick" (we use an OCaml interface for compilation, and inject at link time the implementation) - Hannes: the MirageOS utility (mirage) emit the required dependencies (e.g. mirage-ptime.solo5) - Hannes: instead of `Ptime.v (PCLOCK.now_d_ps ())`, we now use `Mirage_ptime.now () : Ptime.t` - Hannes: it is a great step forward, once we have pushed these changes through, a next step may be to use the same mechanism for Mirage_net - The user site is best viewed here: https://github.com/mirage/mirage-skeleton/pull/407/files?w=1 - Reynir: there's a change for unikernels without arguments, now a magic "()" argument will be provided (so no need to depend on `noop @-> job` or `~extra_deps:[ dep noop ]`) - Hannes: interested in feedback, and whether it will be convenient for unikraft as well - Sam: it'll be fine, no surprises - Pierre: really appreciate the shape of mirage-skeleton with the above PR, the unikernels are very readable - Pierre: is there a difference in the binary size? - Hannes: not sure, haven't measured it - Hannes: every unikernel now depends on the mirage-crypto-rng (but we can manually remove it with `~random:no_random`), not sure whether we should optimize for the binary size of hello world ### MirageOS retreat in May (13th - 20th) - https://retreat.mirageos.org - please sign up and spread the word ### Qubes-mirage-firewall - Pierre will cut a release - Thereafter, likely defunctorization & OCaml 5 support in the next release ### Next meeting: Feb 24th 10:00 - 12:00 CET ## Meeting January 27th 10:00 - 12:00 CET - Participants: Pierre, Sam, Reynir, Pixie ### GC compactor - The GC compactor will unmap _ leading to ABORT - As you have to call the compactor explicitly we probably did not test it - Maybe we don't actually need compaction? - Sam: Observeration from mirage-www: - the behavior was similar to ocaml 4 version - if we tweak space overhead to 80 we get a similar profile as ocaml 4 version - Pierre: we can't deny users to call `Gc.compact ()` which would lead to a crash - Pierre: implementing mmap/munmap would be a huge effort - Sam: https://github.com/ocaml-flambda/flambda-backend/pull/3179 propose a different version of the compactor that doesn't fragment memory - Sam: So we could switch the whole GC (but this is a lot of work) - Pierre: Maybe time to investigate another allocator (e.g. jemalloc) - Pierre: everytime I looked into this it was very difficult - Sam: is writing or switch the allocator difficult? - Pierre: switching is easy, writing one is hard. Writing an allocator based on mmap/munmap is easy, the other way is hard. - Sam: we can imagine something radical like we know how much we will allocate to the unikernel. - Pierre: we know that: all the memory at the beginning. It's in sysdeps_solo5.c. We have `_nolibc_init()` where we can init everything. - Pierre: doesn't have time, but can look into a list of allocators - Pierre: We want to have very fast free memory estimation like we do in Qubes firewall. Will check that. - Sam: we do many small allocations in Mirage unikernels? - Pierre: we do as long as we have BigArrays - Pierre, with Sam agreeing: in the meantime we should encourage users to not call `Gc.compact ()` ### Performance: bytes vs. bigarray (cstruct) - Pierre: No news :D - Sam: I agree about reducing small (bigarray) allocations cf. previous item. ### Unikraft - Sam: Unikraft has real mmap/munmap which would be another way to solve the GC compactor issue - Pierre: is it possible to "steal" the allocator from unikraft? Also wrt. licensing. - Sam: Re: licensing, it may be quite a mess. In the code base it's already quite a mix of licenses: GPL, MIT, ... - Sam: also possible their implementation uses part of musl - Sam: all in all, not so simple ## Meeting January 13th 10:00 - 12:00 CET - Participants: Pierre, Hannes, Reynir, Sam, Pixie ### Performance: bytes vs. bigarray (cstruct) - Pierre did some performance benchmarks. The results are confusing. - VirtIO and spt are different: in virtio kvm/qemu does potential batching and queuing of packets; spt it is solo5 copying one-by-one - reynir suggests to rewrite the C relay one-to-one in OCaml and take a look at the performance, figuring what the OCaml runtime costs are - best performance is the Linux with a bridge between the two interfaces, that's our goal to compare numbers with -- it is as well a virtual machine with two interfaces doing the copying in kernel space -- MirageOS shouldn't be slower - how/which profiling could we run there? statmemprof (https://github.com/robur-coop/memtrace-mirage -- only covers allocations, no computation (such as CRC)), landmarks (https://github.com/LexiFi/landmarks), SPT: we could also use `perf` utility from Linux - it is still not very clear what is the bottleneck ### Building cross-compilers / OCaml 5 support - Sam: they're merged in OCaml trunk - Sam: backported to OCaml 5.3, plans to update ocaml-solo5 with 5.3 support - Sam: some of the patches are only for ocaml-solo5 - Hannes: we can as well remove the 5.2.1 patches from ocaml-solo5 and move ahead for 5.3 -- if someone wants to usse 5.2.1 they can use the released ocaml-solo5 (1.0.0/1.0.1) - Sam: the good news is that for OCaml 5.4 there will be really few patches needed ### maintenance-intent for MirageOS libraries - unbounded growth of opam repository is unsustainable. Hannes started effort to prune uninstallable, broken and/or unmaintained packages. The `x-maintenance-intent` flag in opam files signals the intent to maintain packages: e.g. only the latest release etc. - hannes main question here is whether there are any reservations of putting x-maintenance-intent: [ "(latest)" ] in the opam files for the MirageOS packages that we maintain (e.g. mirage, mirage-solo5, ...) - pierre and reynir agree with putting the latest there (apart from ocaml-solo5) - sometimes we have big breaking API changes where we likely want to keep the old and new versions (e.g. in ocaml-solo5 we have [ "(latest)" "0.8.(latest)" ]) - we can do that on a package-by-package basis when we do major breaking changes ### Defunctorise MirageOS - hannes: don't want to pass "PCLOCK" "MCLOCK" "TIME" as functors - hannes: the question is about naming, and about requiring the ptime/mtime libraries directly - hannes: there's the opam package called mirage-clock (Mirage_clock), there we have a PCLOCK and a MCLOCK - hannes: currently my suggestion is to name it Mirage_ptime and Mirage_mtime? - hannes: and then the TIME interface should we rename it to Mirage_sleep? Mirage_sleep.sleep? Mirage_sleep.doit? Mirage_sleep.ns! Currently: ```OCaml (** Sleep operations. *) module type S = sig val sleep_ns: int64 -> unit Lwt.t (** [sleep_ns n] Block the current thread for [n] nanoseconds, treating the [n] unsigned. *) end ``` ### MirageOS and unikraft - Pierre is curious what the state thereof is - Sam: work is ongoing, some on private branches ### Meeting time is good? - Sam mentions that the slot is good for him, but every time he is busy with some other thing ### Mollymawk - robur got a grant for working on mollymawk from NLnet / EU tax payers - we're excited to continue the work on it, and this will make (one path for) deployment of MirageOS unikernels easier ### Next meeting in two weeks, on Jan 27th ## Meeting Dec 9th 10:00 - 12:00 CET - Participants: Sam, Hannes, Reynir, Pierre, Pixie ### OCaml 5 - ocaml-solo5 1.0.0 is released, now we can get MirageOS unikernels with OCaml 5.2.1 - TODO we should announce this on discuss.ocaml.org and on mirageos.org - Romain opened a PR for ocaml-solo5 https://github.com/mirage/ocaml-solo5/pull/144 - Pierre: pthread_cond_init is a stub, so pthread_cond_destroy could as well be - Hannes: the second commit is to not use opam-monorepo -- let us briefly remind ourselves that a MirageOS unikernel has a custom libc, so we need to cross-compile all opam packages that include C stubs. The MirageOS 4 way of doing that is to download (using opam-monorepo all opam packages into a "duniverse" directory), and compile everything. - Hannes: it is not entirely clear to me how to detect opam packages that include C stubs (to cross-compile them), but removing the opam-monorepo dependency would be great - Sam: we all agree that if we're cross-compiling to a different architecture, this won't work - Reynir: does cross-compilation work at the moment? Sam: I have tested it, from x86_64 to aarch64 - running on Raspberry PI - Hannes: that cross-compilation is Linux-only so I never tested it - Sam: What do you mean it's Linux-only? - Hannes: _ says it's Linux-only and I never bothered looking into the details. Actually, maybe it's not. -- Turns out the solo5-cross-aarch64 is Linux only - Reynir: maybe there should be an opam flag "this does not include C stubs", but that'd be incomplete and require some bookkeeping. If we mark common packages as such we can avoid some re-fetching. - Hannes: we would need to convince a lot of package maintainers to include this flag. - Hannes: META file includes information if we should link to external archives. Maybe a path forward. Feels a bit hairy, and only works with installed packages and not packages we don't have installed yet. - Hannes: If we separate pthread_cond_destroy() into another PR we can merge that now. For the rest of the PR we can discuss on the mailing list etc. - Sam: I agree. ### What is Mirage CI? - https://github.com/ocurrent/mirage-ci - Likely it has somewhere OCaml 4 hardcoded, and that needs to be changed ### OCaml 5 used on mirageos.org - Sam: The OCaml 5 unikernel uses much more memory - Sam talked with the OCaml GC people - they need more tests - Sam: Now running with different space_overhead parameters - from 120 (default) to 60 - Sam: Currently OCaml pre-allocates lots of pools, it may be sensible to reduce the maximum size of the objects that pools are allocated for -- for MirageOS unikernels with a low memory footprint - Sam: On mirageos.org we have quite some memory available for the unikernels, would be interesting to see how such a unikernel behaves if it has not so much memory - Hannes: you could tweak the deployment shell scripts to give the unikernel less memory - Hannes: is there an issue tracking this on the ocaml/ocaml repository? - Sam: there have been discussions about frama-c and coq and their memory usages with OCaml 5 ### Performance - Pierre didn't have time to test VirtIO further (and won't for the next 3 weeks) ## Meeting Nov 25th 10:00 - 12:00 CET - Participants: Sam, Reynir, Hannes, Pierre ### OCaml 5 - https://github.com/mirage/ocaml-solo5/pull/134 - Sam continued and rebased that PR - the strerror is actually used, it was marked as TODO (we need to implement it) # - Virgile restarted last week the measures on the Mirage website, but we don't have access to the Grafana dashboard - Hannes reviewed some bits and pieces of the ocaml-solo5 PR - The ocaml-src package relies on an extra file, META, that has been moved to opam-source-archives - There could either be a "echo ... > META" or a META.in file for the ocaml-src package, hannes will take a look and propose PRs -- an empty META file may be sufficient - TODO: rebase the patches on 5.2.1 - question: what about the LICENSE if we take values (and comments) from an existing project -- maybe we should copy the entire errno.h? ### unikraft - there's something that works with networking (didn't look into block devices yet) ### Results from performance experiments - Pierre did some benchmarks https://github.com/palainp/simple-fw/tree/no-cstruct - running with solo5-spt as a Qubes guest | Av. bandwitdh (Gbits/sec) | Ocaml 4.14.1 | Ocaml 5.2 | C relay | |---------------------------|--------------------|-----------------|-------------| | Cstruct as BA | 1.16 | 1.12 | 1.58 | | Cstruct as bytes | 1.42 | 1.41 | 1.58 | - running with separate VMs (iperf client, bridge/firewall, iperf server) resulted in the C implementation being slower than the MirageOS ones - TODO someone else reproducing the benchmarks on a separate infrastructure - TODO optimize the C implementation (?) - We can use a bridge interface (brctl) with both network interfaces (so we don't copy from kernel space to user space) - TODO try with virtio - Sam: maybe use sendfile? (that'd be an optimization for the C relay) - To conclude, moving Cstruct from bigarray to bytes improves the performance - Hannes likes the simple-fw performance setup for the lower layers of the network stack, we can learn a lot: different backends, flambda2, trying out queueing (instead of one VMEXIT per packet send/recv) - The remaining blocker for cstruct-as-bytes is the xen backend which relies on non-moving page-aligned memory being passed around. We should continue to work on adapting that code (to use io-page) - Pierre also mentioned that we can look deeper into what allocations we can remove to improve the performance ## Meeting Nov 12th 11:00 - 12:45 CET - Topic: (Hannes, Pierre) cstruct and performance (see e.g. https://github.com/mirage/mirage-net/pull/25) - Participants: Pierre, Sam, Hannes, Anh #### Minutes - Anh is a PhD student (advised by others & Pierre) who works on opportunistic firewalling - using MirageOS as a firewall - Also wants to work on an IDS in MirageOS, similar to suricata - Pierre is an associate professor at University of Rennes, MirageOS core team - mainly works on the qubes-mirage-firewall - joined MirageOS since he uses QubesOS on his main laptop - Hannes is working full-time on MirageOS - Works in Robur Coop with other peoples - Push MirageOS into production - Works on various applications for MirageOS (VPN, dnsmasq, ...) - Has done some performance investigations, and would like to improve the performance of MirageOS to convince more people to use it - Sam works at Tarides since ~2 years - since last year works on MirageOS - mainly to get MirageOS working on OCaml 5 - also to get MirageOS working on unikraft (replacing solo5) - since solo5 lacks a bit maintenance, also performance (unikraft has batch IO), maybe some day also multicore - Cstruct & performances - Cstruct are important for some backends (Xen) where non moving memory areas are shared amoing domains - Cstruct are heavy to allocate (using dlmalloc, is expensive), and this is against the GC (only the finalizer is used for free) - Some work in (e.g. mirage-crypto) has shown performance improvments (2.5x - 3x) during the Cstruct->{string, bytes} swap - Sam: for some operations we would need to copy - Sam: for a packet receive / send ring, we need non-moving memory as well - Pierre: in the qubes-mirage-firewall we do a lot of copies anyways, e.g. NAT - Pierre: it is "probably"(TM) not too painful to move from Cstruct.t to bytes - Pierre: a big issue is the finalizer, it is unclear when it is called, and the memory is fragmented a lot - Hannes: API-wise, mirage-net receive function takes a callback (Cstruct.t -> unit) Lwt.t, where the mirage-net allocated a buffer and passes it to the callback - Hannes: and the send function gets a `size` hint, and allocates a buffer to be filled - Hannes: what about ownership? Should the mirage-net receive, once the callback has finished, reclaim ownership and reuse the piece of memory? - Pierre: maybe an opaque type is the path to go? - Pierre: should the send be a write-only buffer, the receive a read-only buffer? -- Hannes: there's Cstruct_cap that uses phantom types for it - Sam: maybe move to an abstract API would help to benchmark the two options, test them on real workloads - Hannes: next to types (API), the question is about ownership (and who is responsible to allocate / free the memory) - Hannes: asking the question the other way around, from the application: what should be done for a packet that is received at the firewall? - Hannes: from my point of view, the perfect firewall should not copy: once a packet is received (given a ring buffer of received packets), this packet should copied (and eventually modified, if NAT needs to be done) to an element of the send ring buffer -- there shouldn't be an allocation of the entire packet in the code - Pierre: we should avoid any allocations, and also all copies - Pierre: started to use a bridge firewall which doesn't copy, and avoiding copies is good for performance (for e.g. solo5), for xen the copy we can't really avoid (on xen, you either need to copy or you would need to reconfigure with which VM you share the memory) - Hannes: with the ring buffer approach, we can't really avoid the copying -- it would mean a lot of buerocracy, and the ownership and lifetime of a buffer in the ring buffer isn't well-specified anymore - Hannes: given xen and uring, the ring buffer would need to contain non-moving memory, for solo5-hvt/spt it shouldn't matter -- but is there a difference between using bytes or bigarray for such a ring buffer? - Hannes also tried to write a library that has an abstract type t and is backed by either byte or bigarray memory (and the implementation can be selected at compile time - so no functorisation, but you get a B.get_uint8 etc. directly), but the issue is that exposing the raw memory from bigarray makes the OCaml runtime unhappy -> segmentation fault - Florian mentioned each OCaml value needs to have a tag/header, so we'd need to allocate a bit memory before the page-aligned stuff, and put the header in there - so that may be a path to investigate - Pierre: not sure how the Cstruct.split works, esp. with the header -> it creates a new OCaml variable with the same starting buffer address, and different offset and length - Hannes: I can see multiple paths to investigate: - virtio firewall using bytes/string vs cstruct - virtio firewall using a ring buffer (i.e. allocate not for each packet) vs allocate for each packet - Hannes: also we (well, Romain) figured that allocations of < 255 bytes is very cheap if you allocate bytes/string (it is in the minor heap) - so the high-performance MirageOS unikernel would allocate data in chunks of < 255 bytes - Hannes: Cstruct is more than the memory region: we have an offset and length as well, thus replacing Cstruct.t by Bytes.t removes some safety, since we don't carry around the offset anymore (and thus the ethernet layer can hardly pass on the payload (Cstruct.shift buf 14)) - Hannes: what is the path forward? do we have a concrete application that we want to use for performance investigations? - Pierre: the qubes-mirage-firewall would be a great study, since there are users (who sometimes complain about the performance) - Pierre: a huge performance benefit in the qubes/xen setting would be segmentation offload - Pierre: started to measure the virtio (simple-fw) with no cstruct (see https://github.com/palainp/simple-fw/tree/no-cstruct (doesn't yet compile, needs some further work on mirage-tcpip)) - Pierre: likes the idea to not trust the upper layer, an abstract type would be great - Hannes: the abstract type could as well be cstruct.t, and have an implementation that uses bytes instead of bigarray.t for switching ;) - Pierre: we had this in 2022, but the qubes-mirage-firewall fails to compile with it - Hannes: let's use that cstruct-backed-by-bytes branch (https://github.com/hannesm/ocaml-cstruct/tree/no-bigarray) and test the simple-fw with virtio on it :) [and for now, ignore the qubes-mirage-firewall compilation issues] #### OCaml 5 and ocaml-solo5 - how should we move? - OCaml 5 has a different GC which memory profile is different - Virgile tested that PR on the mirage website, redirecting every other flow to the OCaml 5 unikernel - The behaviour was different under lots of stress (with aborted connections) -- it would be interesting to see whether under normal conditions there's a difference? - With OCaml 5.3, there had been various GC fixes, and big users like Coq/Frama-C, maybe time to look into it again - With OCaml 5, we need to call GC.compact manually - Pierre: it compiles fine for the qubes-mirage-firewall, but doesn't have any long-term runs (only ~10 hours) - with a slightly improved memory bandwidth - Pierre: also tested on dns-resolver, which died due to memory fragmentation (so we should move the bytes) - Pierre: with OCaml 5, it uses more memory at startup - Pierre plans to test simple-fw with bytes x bigarray on OCaml 4 x OCaml 5 - Sam: we should move forward to test it in real conditions - Sam: we should merge and release, maybe something like the ocaml compiler with release candidates - so it is available, but you've to ask for it explicitly - Hannes: maybe not even needed, since ocaml-solo5 depends on the OCaml version of your switch, and so if you're using OCaml 5, you'll get the ocaml-solo5 compatible with OCaml 5, and if you're using OCaml 4, you'll get the ocaml-solo5 compatible with OCaml 4 #### next meeting, Mon 25th at 10:00 CET - ~12:00 CET