Using BigBlueButton properly

Bonjour,

In the past two months I attended audio only team meetings on a self-hosted Jitsi server (using a 2GB RAM server, the cheapest I could find). The experience and the audio quality was good despite participants having various connection problems (ranging from poor ADSL to poor 4G).

After discovering the difference between SFU and MCU, I was however highly motivated to try BigBlueButton and upgrade to having video in addition to audio. And dozens of participants instead of ten. So I went ahead, installed BBB and organized the team meeting on this server instead of Jitsi the next week. But it was not a good experience because:

  • there were micro gaps (less than a second) in the sound
  • cracks could be heard, sometime so frequently that the conversation was disturbed
  • occasionally the sound was so distorted that the person had to repeat

Removing the video feeds did not improve the situation and there was less than ten of us.

I was not discouraged because people I know and trust reported using BBB successfully with video and a large number of participants (sometime over 100). I soon discovered that the machine I provisioned for the job was too small. But upgrading did not fix the problem. Looking a bit more it appeared that I was a victim of a semi-random bug that nobody seemed to fully understand.

It started in March 2020 and was still open at the time, with many reports similar to mine and also many reports of people saying they could not repeat the problem, developers included. A consensus developed around the idea that the root cause of the bug was the ancient version of Freeswitch included in BBB. It was upgraded and the bug was closed.

I was tracking this bug, patiently waiting for it to be resolved and was happy to resume using BBB. This time, instead of installing my own server, I had the opportunity to use a server known to successfully host meetings with over 100 people. And the next week, the team meeting happened on this server. Much to my surprise the problems were not as severe but they still persisted. Extracts of the recording of the meeting demonstrate:

Cracks
Inaudible voice
Micro gap

Interestingly the micro gap is not that disturbing because it is so short. However, once you noticed it is difficult to forget about it. In the 2 second sequence above, it is audible in the avoir word and can be seen in a mixer. The light blue sound should be one word but is cut during a fraction of a second.

mixer

I have a theory that would explain why some people have a good experience and others keep having the problems described above. It could simply be because the BBB server does not mix audio properly when one of the speakers has a connection that either has high latency (>5ms) or poor bandwidth (occasionally dropping below 10Mb/s). In all the success stories I’ve read, the speakers (those broadcasting audio and video) presumably had an excellent connectivity. And the audience was muted by the moderators. It’s difficult to know for sure because none of the success story are specific about that particular constraint.

Although 150+ participants consume significant resources on the server (i.e. around 12 threads), I’ve been told that a 40+ participant meeting was held successfully on a virtual machine with 4 cores on an otherwise rather occupied server. And since the server on which the last team meeting was held had much more hardware resources than necessary, I reckon this is not the root of the problem.

It could simply be that since half of the participants have a poor connection, it creates the conditions for the BBB server to do poorly when mixing audio. When I get a chance I’ll try to conduct a team meeting with all participants having excellent connectivity and report my findings here.

To be continued!

Yesterday the April general assembly was held online on a BBB instance provided by Octopuce. It lasted about three hours with ~50 participants. The primary motivation for writing this post-mortem focused on technical problems is to emphasize that they are to be expected and will not disrupt the meeting, although they are never mentioned in the BBB success stories. The first time I ran into them I was taken aback because I never heard of them and thought something was wrong with how BBB was used or, worse, that they were a sign that in the middle of the meeting they could suddenly get worse and prevent the meeting from happening. I’m relieved to share this excellent experience. In a nutshell, it confirms that the conditions of success are:

  • All participants are muted / camera off on entry
  • All speakers are behind an excellent internet connection

april

The meeting was divided in three phases:

  • The general assembly for the most part with one or two speakers, all other participants muted and not allowed to share their camera on entry. An audio test was done with all speakers before the meeting. The April staff was in their offices, with an excellent internet connection. The executive director coordinated the speakers and was briefed to replace them in case they were not available at the last minute (it happened once).
  • After the general assembly all participants were invited to activate their webcam.
  • After the webcam activation all participants were invited to activate their mic.

General assembly

The audio and webcam tests with the speakers was useful: it was an opportunity to fix various problems on their side. Nothing specific to BBB and the most common issue was an echo because they were not using a headset.

Micro cracks and gaps: When participants joined, music was playing, broadcasted by a participant on purpose. The sound was not good but it was helpful because it gave an audio confirmation to everyone. The gaps and cracks mentioned in this topic could be heard and some people asked if they were the only one hearing them. When music is playing, they are sometime more noticeable than when someone is speaking.

2 out of 50 gave up connecting: A handful of participants (out of 50) had technical problems and two of them gave up trying to fix them. In my opinion the problem was on their side and they would have been fixed eventually but they were frustrated and did not want to spend more time trying. In two cases the problem was fixed by reloading the browser tab running BBB and nothing else. There was not enough clues to produce a bug report.

Private chat is not noticed: To debug the technical problems of the participants, the private chat was used, after notifying the participant to join this private chat in the general chat window. Trying to catch the attention of someone using the private chat window notification (i.e. typing a phrase in the private chat window) always failed because no-one seems to see or understand the meaning of the red circle in the chat list. I reckon most participants were not familiar with the user interface of BBB.

Slow chat: In the beginning the general chat was responsive but as the number of participants grew it became slower. On my laptop (Intel® Core™ i7-7600U CPU @ 2.80GHz and Firefox 77) there was a lag when typing (from one second to five seconds before the characters appear) and another lag after sending the message (from 5 to 15 seconds before the sentence showed). Other participants reported the chat to be slow as well but I’m not sure if their experience was worse or better than mine.

Slow or misconfigured ADSL: I first tried to connect with an ADSL connection that was not good, even though I was the only one using it. After trying a few times and consistently getting a 1007 ICE… Failed joining audio. error. But I did not have time to investigate and switched to a LTE connection instead.

Camera and mic activation

Simultaneous activation of all cameras reboot the multimedia server: During about 15 minutes, at the end of the meeting, volunteers were asked to activate their camera. About 25 of them did and it triggered a reboot of the multimedia server after less than 30 seconds. It came back a few seconds later and automatically reconnected all participants, one every 5 seconds or so. They were able to virtually share a beer as shown in the screenshot above. It suggests that all would have been fine if the participants were allowed to activate their camera when joining the meeting instead of doing it all at once.

Audio mixer failure: the discussion with the cameras on had more micro cracks and gaps than before but nobody noticed. Most of the time only one person at a time spoke and nobody asked them to repeat. There were occasional laughs and interruptions and the audio mixer failure (i.e. the output cannot be understood ask described in the topic message) can be heard. But it was not disruptive, probably because nobody cared to loose a little bit of the informal conversation. Just like nobody is upset when they don’t understand 100% of what is said during a beer event IRL. But that’s just a wild guess on my part.