Back

Technologies:

javascriptjavascript
avatar
Tolerim
a month ago

The MediaStream is not properly blending audio.

As a new developer, I'm trying to record both system and mic audio while capturing the screen using mediaDevices.getDisplayMedia for the screen and system audio, and mediaDevices.getUserMedia for the mic audio. I'm mixing the streams with "mixedStream = new MediaStream([...stream.getTracks(), ...audio.getTracks()]);" However, only the mic audio is captured. According to my research, when "audio" is set to true in getDisplayMedia, only system audio is captured, and when "audio" is set to true in getUserMedia, only mic audio is captured. If both are set to true, only system audio is captured. I'm new to these APIs, can anyone check the code in the Github link provided (text in dist folder main.js file) and tell me what I'm doing wrong? My ultimate goal is to record both the screen with system audio (tab/all) and mic audio (either from the system mic or plugged earphones) using JavaScript. To enclose the code I'm sharing, use 
 tags. Happy coding!
Answers(2)
avatar
Vogi
a month ago
Verified Answer
I understand that you are trying to record both system and mic audio while capturing the screen using mediaDevices.getDisplayMedia for the screen and system audio, and mediaDevices.getUserMedia for the mic audio in JavaScript. According to your research, when "audio" is set to true in getDisplayMedia, only system audio is captured, and when "audio" is set to true in getUserMedia, only mic audio is captured. If both are set to true, only system audio is captured.
I found a StackOverflow post that might help you with your problem. The post suggests adding the audio tracks from the microphone stream to the display stream using the MediaStream.addTrack() method. Here's an example code snippet from the post:

const stream = await navigator.mediaDevices.getDisplayMedia({ video: true, audio: true });
const audioStream = await navigator.mediaDevices.getUserMedia({ audio: true });
audioStream.getAudioTracks().forEach(track => {
  stream.addTrack(track);
});

I hope this helps! Let me know if you have any other questions.
Sources
How to record your screen with audio (System Audio, Mic)javascript - getDisplayMedia does not record all audio output from my ...4 Ways to Record a Screen on Windows 10 with AudioSharex record screen audio along with microphone - Super UserUsing the MediaStream Recording API - Web APIs | MDN - Mozilla DeveloperMediaDevices: getDisplayMedia() method - Web APIs | MDN
avatar
Tolerim
a month ago
The issue you are facing is that when both audio options are set to true in navigator.mediaDevices.getDisplayMedia() and navigator.mediaDevices.getUserMedia(), only the system audio is captured. To capture both system and mic audio, you can try using the navigator.mediaDevices.enumerateDevices() method to get a list of available audio input devices and select the appropriate one for navigator.mediaDevices.getUserMedia(). Then, you can use MediaStreamTrack methods like addTrack() and removeTrack() to dynamically add/remove audio tracks to/from the MediaStream being recorded. Here's an example implementation:

async function setupStream() {
  try {
    // Get video and system audio media
    const screenStream = await navigator.mediaDevices.getDisplayMedia({
      video: true,
      audio: { // Only get system audio
        mandatory: {
          chromeMediaSource: 'desktop',
          chromeMediaSourceId: 'screen:0',
        },
      },
    });

    // Get mic audio media
    const audioDevices = await navigator.mediaDevices.enumerateDevices();
    const micDevice = audioDevices.find(device => device.kind === 'audioinput');
    const micStream = await navigator.mediaDevices.getUserMedia({
      audio: {
        deviceId: micDevice.deviceId, // Use selected mic device
      },
    });

    // Merge audio and video streams
    mixedStream = new MediaStream([...screenStream.getTracks(), ...micStream.getTracks()]);

    // Setup video feedback
    setupVideoFeedback();
  } catch (err) {
    console.error(err);
  }
}

async function startRecording() {
  await setupStream();

  if (mixedStream) {
    // Start recording mixed stream
    recorder = new MediaRecorder(mixedStream);
    recorder.ondataavailable = handleDataAvailable;
    recorder.onstop = handleStop;
    recorder.start(1000);

    // Disable start button, enable stop button
    startButton.disabled = true;
    stopButton.disabled = false;

    console.log('Recording started');
  } else {
    console.warn('No stream available');
  }
}

function stopRecording() {
  // Stop recording and cleanup
  recorder.stop();
  mixedStream.getTracks().forEach(track => track.stop());

  // Disable stop button, enable start button
  startButton.disabled = false;
  stopButton.disabled = true;

  console.log('Recording stopped');
}

In this example, we separate the audio and video streams and only get the system audio for the getDisplayMedia() call, then use enumerateDevices() to get a list of audio input devices and select the appropriate one for the getUserMedia() call. We then dynamically merge the audio and video streams using MediaStreamTrack methods and pass the resulting MediaStream to the MediaRecorder constructor.
Note that you may need to adjust the chromeMediaSourceId property in the audio options of getDisplayMedia() to specify which audio stream to capture (e.g. tab, system, etc.) depending on your OS and browser.
;