使用Android的MediaCodec和MediaMuxer对AAC音频进行Muxing

Question

我正在修改一个Android框架示例，将MediaCodec生成的基本AAC流打包成一个独立的。mp4文件。我使用的是一个MediaMuxer实例，其中包含一个由MediaCodec实例生成的AAC曲目。

然而，我总是最终得到一个错误消息调用mMediaMuxer.writeSampleData（trackIndex， codedData， BufferInfo）：

E/MPEG4Writer﹕时间戳0

当我在mCodec中将原始输入数据排队时。queueInputBuffer（…） 我为每个框架示例提供了0作为时间戳值（我也尝试过使用单调递增的时间戳值来获得相同的结果。我用同样的方法成功地将原始相机帧编码为h264/mp4文件）。

查看完整的来源

最相关的片段：

private static void testEncoder(String componentName, MediaFormat format, Context c) {
    int trackIndex = 0;
    boolean mMuxerStarted = false;
    File f = FileUtils.createTempFileInRootAppStorage(c, "aac_test_" + new Date().getTime() + ".mp4");
    MediaCodec codec = MediaCodec.createByCodecName(componentName);

    try {
        codec.configure(
                format,
                null /* surface */,
                null /* crypto */,
                MediaCodec.CONFIGURE_FLAG_ENCODE);
    } catch (IllegalStateException e) {
        Log.e(TAG, "codec '" + componentName + "' failed configuration.");

    }

    codec.start();

    try {
        mMediaMuxer = new MediaMuxer(f.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
    } catch (IOException ioe) {
        throw new RuntimeException("MediaMuxer creation failed", ioe);
    }

    ByteBuffer[] codecInputBuffers = codec.getInputBuffers();
    ByteBuffer[] codecOutputBuffers = codec.getOutputBuffers();

    int numBytesSubmitted = 0;
    boolean doneSubmittingInput = false;
    int numBytesDequeued = 0;

    while (true) {
        int index;

        if (!doneSubmittingInput) {
            index = codec.dequeueInputBuffer(kTimeoutUs /* timeoutUs */);

            if (index != MediaCodec.INFO_TRY_AGAIN_LATER) {
                if (numBytesSubmitted >= kNumInputBytes) {
                    Log.i(TAG, "queueing EOS to inputBuffer");
                    codec.queueInputBuffer(
                            index,
                            0 /* offset */,
                            0 /* size */,
                            0 /* timeUs */,
                            MediaCodec.BUFFER_FLAG_END_OF_STREAM);

                    if (VERBOSE) {
                        Log.d(TAG, "queued input EOS.");
                    }

                    doneSubmittingInput = true;
                } else {
                    int size = queueInputBuffer(
                            codec, codecInputBuffers, index);

                    numBytesSubmitted += size;

                    if (VERBOSE) {
                        Log.d(TAG, "queued " + size + " bytes of input data.");
                    }
                }
            }
        }

        MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
        index = codec.dequeueOutputBuffer(info, kTimeoutUs /* timeoutUs */);

        if (index == MediaCodec.INFO_TRY_AGAIN_LATER) {
        } else if (index == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
            MediaFormat newFormat = codec.getOutputFormat();
            trackIndex = mMediaMuxer.addTrack(newFormat);
            mMediaMuxer.start();
            mMuxerStarted = true;
        } else if (index == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
            codecOutputBuffers = codec.getOutputBuffers();
        } else {
            // Write to muxer
            ByteBuffer encodedData = codecOutputBuffers[index];
            if (encodedData == null) {
                throw new RuntimeException("encoderOutputBuffer " + index +
                        " was null");
            }

            if ((info.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
                // The codec config data was pulled out and fed to the muxer when we got
                // the INFO_OUTPUT_FORMAT_CHANGED status.  Ignore it.
                if (VERBOSE) Log.d(TAG, "ignoring BUFFER_FLAG_CODEC_CONFIG");
                info.size = 0;
            }

            if (info.size != 0) {
                if (!mMuxerStarted) {
                    throw new RuntimeException("muxer hasn't started");
                }

                // adjust the ByteBuffer values to match BufferInfo (not needed?)
                encodedData.position(info.offset);
                encodedData.limit(info.offset + info.size);

                mMediaMuxer.writeSampleData(trackIndex, encodedData, info);
                if (VERBOSE) Log.d(TAG, "sent " + info.size + " audio bytes to muxer with pts " + info.presentationTimeUs);
            }

            codec.releaseOutputBuffer(index, false);

            // End write to muxer
            numBytesDequeued += info.size;

            if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
                if (VERBOSE) {
                    Log.d(TAG, "dequeued output EOS.");
                }
                break;
            }

            if (VERBOSE) {
                Log.d(TAG, "dequeued " + info.size + " bytes of output data.");
            }
        }
    }

    if (VERBOSE) {
        Log.d(TAG, "queued a total of " + numBytesSubmitted + "bytes, "
                + "dequeued " + numBytesDequeued + " bytes.");
    }

    int sampleRate = format.getInteger(MediaFormat.KEY_SAMPLE_RATE);
    int channelCount = format.getInteger(MediaFormat.KEY_CHANNEL_COUNT);
    int inBitrate = sampleRate * channelCount * 16;  // bit/sec
    int outBitrate = format.getInteger(MediaFormat.KEY_BIT_RATE);

    float desiredRatio = (float)outBitrate / (float)inBitrate;
    float actualRatio = (float)numBytesDequeued / (float)numBytesSubmitted;

    if (actualRatio < 0.9 * desiredRatio || actualRatio > 1.1 * desiredRatio) {
        Log.w(TAG, "desiredRatio = " + desiredRatio
                + ", actualRatio = " + actualRatio);
    }


    codec.release();
    mMediaMuxer.stop();
    mMediaMuxer.release();
    codec = null;
}

更新：我发现了一个我认为存在于MediaCodec中的根本症状。：

我将presentationTimeUs=1000发送到queueInputBuffer（…） 但接收信息。调用MediaCodec后，presentationTimeUs=33219。出列输出缓冲区（info，timeoutUs）。法登就这一行为发表了有益的评论。




Answer 1

问题发生是因为您无序地接收缓冲区：尝试添加以下测试：

if(lastAudioPresentationTime == -1) {
    lastAudioPresentationTime = bufferInfo.presentationTimeUs;
}
else if (lastAudioPresentationTime < bufferInfo.presentationTimeUs) {
    lastAudioPresentationTime = bufferInfo.presentationTimeUs;
}
if ((bufferInfo.size != 0) && (lastAudioPresentationTime <= bufferInfo.presentationTimeUs)) {
    if (!mMuxerStarted) {
        throw new RuntimeException("muxer hasn't started");
    }
    // adjust the ByteBuffer values to match BufferInfo (not needed?)
    encodedData.position(bufferInfo.offset);
    encodedData.limit(bufferInfo.offset + bufferInfo.size);
    mMuxer.writeSampleData(trackIndex.index, encodedData, bufferInfo);
}

encoder.releaseOutputBuffer(encoderStatus, false);

Answer 2

上面回答https://stackoverflow.com/a/18966374/6463821的代码还提供了timestampUs XXX

所以我对这个问题的解决方案是生成第一个timstamp，每一个下一个样本都会根据样本的持续时间增加时间戳（取决于比特率、音频格式、通道配置）。

BUFFER_DURATION_US = 1_000_000 * (ARR_SIZE / AUDIO_CHANNELS) / SAMPLE_AUDIO_RATE_IN_HZ;

...

long firstPresentationTimeUs = System.nanoTime() / 1000;

...

audioRecord.read(shortBuffer, OFFSET, ARR_SIZE);
long presentationTimeUs = count++ * BUFFER_DURATION + firstPresentationTimeUs;

从音频记录中读取应在单独的线程中，并且所有读取缓冲区应添加到队列中，而无需等待编码或任何其他操作，以防止丢失音频样本。

worker =
        new Thread() {

            @Override
            public void run() {
                try {

                    AudioFrameReader reader =
                            new AudioFrameReader(audioRecord);

                    while (!isInterrupted()) {
                        Thread.sleep(10);

                        addToQueue(
                                reader
                                        .read());
                    }

                } catch (InterruptedException e) {
                    Log.w(TAG, "run: ", e);
                }
            }
        };

Answer 3

匿名用户

感谢fadden的帮助，我在Github上获得了概念验证音频编码器和视频音频编码器。总结：

将AudioRecord的样本发送到MediaCodecMediaMuxer包装器。使用音频编码>系统录制。读（…）可以很好地用作音频时间戳，前提是您的轮询次数足够频繁，以避免填满AudioRecord的内部缓冲区（以避免在调用read的时间和AudioRecord记录样本的时间之间发生漂移）。太糟糕了录音不能直接传达时间戳。。。

// Setup AudioRecord
while (isRecording) {
    audioPresentationTimeNs = System.nanoTime();
    audioRecord.read(dataBuffer, 0, samplesPerFrame);
    hwEncoder.offerAudioEncoder(dataBuffer.clone(), audioPresentationTimeNs);
}

请注意，AudioRecord仅保证支持16位PCM样本，尽管MediaCodec。queueInputBuffer将输入作为字节[]。将字节[]传递给录音。读（数据缓冲，…） 威尔

我发现用这种方式轮询仍然偶尔会生成一个timestampUs XXX