iOS音频编程之混音

iOS音频编程之混音

需求:多个音频源混合后输出,

项目说明:项目中采样4路音频源混合,音频源包含44100hz采样率,3000hz采样率,单声道和立体声;使用MixerVoiceHandle封装混音处理,用户只需要初始化音频文件路径数组,调用启动混音接口,就可实现多路音频混合输出

AVAudioSession设置

没有把对AVAudioSession的设置封装进MixerVoiceHandle中,用户的app可能对Session会有不同的设置(如录音),对于混音只要保证session能播放,bufferDuration和采样率为MixerVoiceHandle申请的一样即可

AVAudioSession *sessionInstance = [AVAudioSession sharedInstance];

[sessionInstance setCategory:AVAudioSessionCategoryPlayback error:&error];
handleError(error);

NSTimeInterval bufferDuration = kSessionBufDuration;
[sessionInstance setPreferredIOBufferDuration:bufferDuration error:&error];
handleError(error);
    
double hwSampleRate = kGraphSampleRate;
[sessionInstance setPreferredSampleRate:hwSampleRate error:&error];
handleError(error);
//接下来设置AVAudioSessionInterruptionNotification和AVAudioSessionRouteChangeNotification,省略

kSessionBufDuration为0.05s,kGraphSampleRate44100hz,
session的IOBufferDuration的意思就是在各Audio Unit的回调函数中提供0.005s时间的数据,如录音时,采集到0.005s的数据会进入一次回调函数

读取音频数据

读取音频数据到内存中,耗时比较久,放到后台线程中执行,而初始化AUGraph时,用到了读取出的音频信息,所以干脆将读取音频数据,混音设置都放在了一个后台的串行队列中。

_mSoundBufferP = (SoundBufferPtr)malloc(sizeof(SoundBuffer) * self.sourceArr.count);

for (int i = 0; i < self.sourceArr.count; i++) {
    NSLog(@"read Audio file : %@",self.sourceArr[i]);
    CFURLRef url = CFURLCreateWithFileSystemPath(kCFAllocatorDefault, (CFStringRef)self.sourceArr[i], kCFURLPOSIXPathStyle, false);
    ExtAudioFileRef fp;
    //open the audio file
    CheckError(ExtAudioFileOpenURL(url, &fp), "cant open the file");
    
    AudioStreamBasicDescription fileFormat;
    UInt32 propSize = sizeof(fileFormat);
    
    //read the file data format , it represents the file's actual data format.
    CheckError(ExtAudioFileGetProperty(fp, kExtAudioFileProperty_FileDataFormat,
                                       &propSize, &fileFormat),
               "read audio data format from file");
    
    double rateRatio = kGraphSampleRate/fileFormat.mSampleRate;
    
    UInt32 channel = 1;
    if (fileFormat.mChannelsPerFrame == 2) {
        channel = 2;
    }
    AVAudioFormat *clientFormat = [[AVAudioFormat alloc] initWithCommonFormat:AVAudioPCMFormatFloat32
                                                                   sampleRate:kGraphSampleRate
                                                                     channels:channel
                                                                  interleaved:NO];
    
    propSize = sizeof(AudioStreamBasicDescription);
    //设置从文件中读出的音频格式
    CheckError(ExtAudioFileSetProperty(fp, kExtAudioFileProperty_ClientDataFormat,
                                       propSize, clientFormat.streamDescription),
               "cant set the file output format");
    //get the file's length in sample frames
    UInt64 numFrames = 0;
    propSize = sizeof(numFrames);
    CheckError(ExtAudioFileGetProperty(fp, kExtAudioFileProperty_FileLengthFrames,
                                       &propSize, &numFrames),
               "cant get the fileLengthFrames");
    
    numFrames = numFrames * rateRatio;
    
    _mSoundBufferP[i].numFrames = (UInt32)numFrames;
    _mSoundBufferP[i].channelCount = channel;
    _mSoundBufferP[i].asbd      = *(clientFormat.streamDescription);
    _mSoundBufferP[i].leftData = (Float32 *)calloc(numFrames, sizeof(Float32));
    if (channel == 2) {
        _mSoundBufferP[i].rightData = (Float32 *)calloc(numFrames, sizeof(Float32));
    }
    
    _mSoundBufferP[i].sampleNum = 0;
    //如果是立体声,还要多为AudioBuffer申请一个空间存放右声道数据
    AudioBufferList *bufList = (AudioBufferList *)malloc(sizeof(AudioBufferList) + sizeof(AudioBuffer)*(channel-1));
    
    AudioBuffer emptyBuffer = {0};
    for (int arrayIndex = 0; arrayIndex < channel; arrayIndex++) {
        bufList->mBuffers[arrayIndex] = emptyBuffer;
    }
    bufList->mNumberBuffers = channel;
    
    bufList->mBuffers[0].mNumberChannels = 1;
    bufList->mBuffers[0].mData = _mSoundBufferP[i].leftData;
    bufList->mBuffers[0].mDataByteSize = (UInt32)numFrames*sizeof(Float32);
    
    if (2 == channel) {
        bufList->mBuffers[1].mNumberChannels = 1;
        bufList->mBuffers[1].mDataByteSize = (UInt32)numFrames*sizeof(Float32);
        bufList->mBuffers[1].mData = _mSoundBufferP[i].rightData;
    }
    
    UInt32 numberOfPacketsToRead = (UInt32) numFrames;
    CheckError(ExtAudioFileRead(fp, &numberOfPacketsToRead,
                                bufList),
               "cant read the audio file");
    free(bufList);
    ExtAudioFileDispose(fp);
}

这段代码就是把音频文件以设置的kExtAudioFileProperty_ClientDataFormat音频格式,读出到_mSoundBufferP数组中

如果您想使用自己准备的音频文件,ExtAudioFileRead读取时返回-50的code,一般是设置读出的目的音频格式(kExtAudioFileProperty_ClientDataFormat)不正确,如源文件是单声道,而想读出的目的格式是立体声

混音设置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
CheckError(NewAUGraph(&_mGraph), "cant new a graph");

AUNode mixerNode;
AUNode outputNode;

AudioComponentDescription mixerACD;
mixerACD.componentType = kAudioUnitType_Mixer;
mixerACD.componentSubType = kAudioUnitSubType_MultiChannelMixer;
mixerACD.componentManufacturer = kAudioUnitManufacturer_Apple;
mixerACD.componentFlags = 0;
mixerACD.componentFlagsMask = 0;

AudioComponentDescription outputACD;
outputACD.componentType = kAudioUnitType_Output;
outputACD.componentSubType = kAudioUnitSubType_RemoteIO;
outputACD.componentManufacturer = kAudioUnitManufacturer_Apple;
outputACD.componentFlags = 0;
outputACD.componentFlagsMask = 0;

CheckError(AUGraphAddNode(_mGraph, &mixerACD,
&mixerNode),
"cant add node");
CheckError(AUGraphAddNode(_mGraph, &outputACD,
&outputNode),
"cant add node");

CheckError(AUGraphConnectNodeInput(_mGraph, mixerNode, 0, outputNode, 0),
"connect mixer Node to output node error");

CheckError(AUGraphOpen(_mGraph), "cant open the graph");

CheckError(AUGraphNodeInfo(_mGraph, mixerNode,
NULL, &_mMixer),
"generate mixer unit error");
CheckError(AUGraphNodeInfo(_mGraph, outputNode, NULL, &_mOutput),
"generate remote I/O unit error");

UInt32 numberOfMixBus = (UInt32)self.sourceArr.count;

//配置混音的路数,有多少个音频文件要混音
CheckError(AudioUnitSetProperty(_mMixer, kAudioUnitProperty_ElementCount, kAudioUnitScope_Input, 0,
&numberOfMixBus, sizeof(numberOfMixBus)),
"set mix elements error");

// Increase the maximum frames per slice allows the mixer unit to accommodate the
// larger slice size used when the screen is locked.
UInt32 maximumFramesPerSlice = 4096;
CheckError( AudioUnitSetProperty (_mMixer,
kAudioUnitProperty_MaximumFramesPerSlice,
kAudioUnitScope_Global,
0,
&maximumFramesPerSlice,
sizeof (maximumFramesPerSlice)
), "cant set kAudioUnitProperty_MaximumFramesPerSlice");
for (int i = 0; i < numberOfMixBus; i++) {
// setup render callback struct
AURenderCallbackStruct rcbs;
rcbs.inputProc = &renderInput;
rcbs.inputProcRefCon = _mSoundBufferP;

CheckError(AUGraphSetNodeInputCallback(_mGraph, mixerNode, i, &rcbs),
"set mixerNode callback error");


AVAudioFormat *clientFormat = [[AVAudioFormat alloc]
initWithCommonFormat:AVAudioPCMFormatFloat32
sampleRate:kGraphSampleRate
channels:_mSoundBufferP[i].channelCount
interleaved:NO];
CheckError(AudioUnitSetProperty(_mMixer,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Input, i,
clientFormat.streamDescription,
sizeof(AudioStreamBasicDescription)),
"cant set the input scope format on bus[i]");
}

double sample = kGraphSampleRate;
CheckError(AudioUnitSetProperty(_mMixer, kAudioUnitProperty_SampleRate,
kAudioUnitScope_Output, 0,&sample , sizeof(sample)),
"cant the mixer unit output sample");

//未设置mixer unit 的kAudioUnitScope_Output的0的音频格式(AudioComponentDescription) 未设置io unit kAudioUnitScope_Output 的element 1的输出AudioComponentDescription

//CheckError(AudioUnitSetProperty(_mMixer, kAudioUnitProperty_StreamFormat,
//kAudioUnitScope_Output, 0, xxxx, sizeof(AudioStreamBasicDescription)), "xxx");

CheckError(AUGraphInitialize(_mGraph), "cant initial graph");

新建AUGraph->新建AUNode(混音Node,音频输出Node)->将混音Node和音频输出Node连接(连接后,混音后的输出流入音频输出的Audio Unit)->从AUNode中得到相应的Audio Unit->设置Mixer Audio Unit的混音路数->设置各路混音的回调函数,输入的音频格式->设置混音个输出采样率->Initialize AUGraph


Audio Unit 这张图片是一个Audio Unit; 相对于混音的Unit(type是`kAudioUnitType_Mixer`,subType是`kAudioUnitSubType_MultiChannelMixer`),我个人理解是这样的 左边是`Mixer Unit`,右边是`Remote I/O Unit`,在`Mixer Unit`的Input Scope下,有多少个Element(Bus),由`kAudioUnitProperty_ElementCount`来设置,并分别为Mixer Unit的`Input Scope`下的各个Element(Bus)设置音频格式和输入回调;将音频源合成到`Mixer Unit`的`Output Scope`的`Element 0`上。

混音输入回调

static OSStatus renderInput(void *inRefCon,
                        AudioUnitRenderActionFlags *ioActionFlags,
                        const AudioTimeStamp *inTimeStamp,
                        UInt32 inBusNumber, UInt32 inNumberFrames,
                        AudioBufferList *ioData)
{
SoundBufferPtr sndbuf = (SoundBufferPtr)inRefCon;

UInt32 sample = sndbuf[inBusNumber].sampleNum;      // frame number to start from
UInt32 bufSamples = sndbuf[inBusNumber].numFrames;  // total number of frames in the sound buffer
Float32 *leftData = sndbuf[inBusNumber].leftData; // audio data buffer
Float32 *rightData = nullptr;

Float32 *outL = (Float32 *)ioData->mBuffers[0].mData; // output audio buffer for L channel
Float32 *outR = nullptr;
if (sndbuf[inBusNumber].channelCount == 2) {
    outR = (Float32 *)ioData->mBuffers[1].mData; //out audio buffer for R channel;
    rightData = sndbuf[inBusNumber].rightData;
}
   
for (UInt32 i = 0; i < inNumberFrames; ++i) {
    outL[i] = leftData[sample];
    if (sndbuf[inBusNumber].channelCount == 2) {
        outR[i] = rightData[sample];
    }
    sample++;
    
    if (sample > bufSamples) {
        // start over from the beginning of the data, our audio simply loops
        printf("looping data for bus %d after %ld source frames rendered\n", (unsigned int)inBusNumber, (long)sample-1);
        sample = 0;
    }
}
sndbuf[inBusNumber].sampleNum = sample; // keep track of where we are in the source data buffer

return noErr;
}

将内存中保存的各路音频数据赋值给回调函数的ioData->mBuffer[x].mData,x=0或1

启动或停止AUGraph

初始化完成后,使用AUGraphStart(_mGraph)启动混音,手机就会输出混合后的音频了;使用AUGraphStop(_mGraph)停止输出。

音量和各路音频使能控制

可以单独控制各路音频的音量(对Mixer UnitInput Scope下的各路ElementkMultiChannelMixerParam_Volume设置音量),也可以控制整体的音量(对Mixer UnitOutput Scope下的Element 0kMultiChannelMixerParam_Volume设置音量);
Mixer UnitInput Scope下的各路ElementkMultiChannelMixerParam_Enable设置使能此路音频信号是否加入到混音中)

代码下载地址
参考资料
参考代码

1