之前各种事情在身,发现好久没更新文章了,临近年末,就把最近做的视频处理相关的内容整理一下吧~
最近在做视频编辑处理相关的开发,其中之一就是音视频合成,需求是用户可以选择将相册中的视频,然后将一段音乐片段加入其中,并可以实时调整视频原声以及添加的音乐音量,最后合成为一个视频。
分析
首先对于视频处理,万能的ffmpeg肯定可以实现,但依赖ffmpeg并用一段magic一样的语句维护扩展都十分有限,对ffmpeg结构不熟悉的话大量c的api也会无从下手,适合熟悉ffmpeg并且对AVFoundation陌生者使用。
其次的最优方案就是AVFoundation了,作为苹果音视频编辑的利器可谓十分强大,官方有一 demo利用AVAudioEngine来实现音频的混音,甚至可以对pcm数据进行编辑,但是缺点也很明显:1 和视频没什么关系,还得启一个AVAudioPlayerNode来播放(那还不如单独用AVAudioPlayer得了) 2 并没有对音频如“美声,变音”之类的需求。所以不作为考虑范围,不过可以实现一些特殊音效还是很厉害的,感兴趣可以下来官方demo-Using AVAudioEngine for Playback, Mixing and Recording (AVAEMixerSample) 看看。
我最后选用的方案就是AVAudioMix,熟悉AVPlayer以及AVPlayerItem的话可能会注意到AVAudioMix 是作为属性存在于AVPlayerItem的分类中。
/*! @property audioMix @abstract Indicates the audio mix parameters to be applied during playback @discussion The inputParameters of the AVAudioMix must have trackIDs that correspond to a track of the receiver's asset. Otherwise they will be ignored. (See AVAudioMix.h for the declaration of AVAudioMixInputParameters and AVPlayerItem's asset property.) */ @property (nonatomic, copy, nullable) AVAudioMix *audioMix;
"Indicates the audio mix parameters to be applied during playback" 表明audioMix是可以在播放的时设置,需要注意的就是trackID需要对应。
补充:可能有人觉得最简单的是同时创建一个AVPlayer负责播放视频,一个AVAudioPlayer播放音乐;当然这种方法是可以实现基本需求,但完美出同步这俩个播放器的状态会是一个问题,而且最终还是要经历混音写文件过程,从逻辑上看十分糟糕。
播放实现
为了表述清晰下面省略AVPlayer等没太大关系的代码,同样也可以下载我的 demo 来查看所有内容。
流程如下:
1 创建视频以及音频的AVURLAsset
AVURLAsset *videoAsset = [AVURLAsset assetWithURL:[NSURL fileURLWithPath:[[NSBundle mainBundle] pathForResource:@"demo" ofType:@"mp4"]]]; AVURLAsset *musicAsset = [AVURLAsset assetWithURL:[NSURL fileURLWithPath:[[NSBundle mainBundle] pathForResource:@"music" ofType:@"mp3"]]];
2 声明并实例化音视频处理的核心类
@property (nonatomic, readwrite, strong) AVMutableComposition *composition; @property (nonatomic, readwrite, strong) AVMutableVideoComposition *videoComposition; @property (nonatomic, readwrite, strong) AVMutableAudioMix *audioMix; AVMutableComposition *composition = [AVMutableComposition composition]; AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoComposition]; AVMutableAudioMix *audioMix = [AVMutableAudioMix audioMix];
3 创建1条视频处理轨道以及2条音频处理轨道(视频原声+添加的音乐这俩条音轨)
AVMutableCompositionTrack *compositionVideoTracks = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid]; AVMutableCompositionTrack *compositionAudioTracks[2]; compositionAudioTracks[0] = [composition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:kCMPersistentTrackID_Invalid]; compositionAudioTracks[1] = [composition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:kCMPersistentTrackID_Invalid];
4 根据之前创建好的视频以及音频资源(AVURLAsset)实例化一条视频轨道以及2条音频轨道
AVAssetTrack *videoTrack = [[asset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0]; [compositionVideoTracks insertTimeRange:self.videoTimeRange ofTrack:videoTrack atTime:kCMTimeZero error:&error]; AVAssetTrack *audioTrack = [[asset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0]; [_comTrack1 insertTimeRange:self.videoTimeRange ofTrack:audioTrack atTime:kCMTimeZero error:&error]; AVAssetTrack *musicTrack = [[self.musicAsset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0]; [_comTrack2 insertTimeRange:self.videoTimeRange ofTrack:musicTrack atTime:kCMTimeZero error:&error];
5 配置AVMutableAudioMix参数,注意这里的trackID一定得是上面创建的AVMutableCompositionTrack对应的trackID,而不是AVAssetTrack中的trackID,之前使用AVAssetTrack出过很奇怪的问题,而后在StackOverFlow上找到了这个解决方案
NSMutableArray<AVAudioMixInputParameters *> *trackMixArray = [NSMutableArray<AVAudioMixInputParameters *> array]; { AVMutableAudioMixInputParameters *trackMix1 = [AVMutableAudioMixInputParameters audioMixInputParametersWithTrack:_comTrack1]; trackMix1.trackID = _comTrack1.trackID; [trackMix1 setVolume:_videoVolume atTime:kCMTimeZero]; [trackMixArray addObject:trackMix1]; AVMutableAudioMixInputParameters *trackMix2 = [AVMutableAudioMixInputParameters audioMixInputParametersWithTrack:_comTrack2]; trackMix2.trackID = _comTrack2.trackID; [trackMix2 setVolume:_musicVolume atTime:kCMTimeZero]; [trackMixArray addObject:trackMix2]; } audioMix.inputParameters = trackMixArray;
6 构建AVPlayerItem, 设置asset以及最重要的audioMix,然后交给AVPlayer就可以同时播放视频与音乐了!
- (AVPlayerItem *)playerItem { if (!_currentItem) { AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:self.composition]; playerItem.videoComposition = self.videoComposition; playerItem.audioMix = self.audioMix; _currentItem = playerItem; } return _currentItem; }
7 播放时调整音量,这里其实和第5步一样,重新配置AVMutableAudioMix参数后赋值给AVPlayerItem,设置音乐音量同理
- (void)setVideoVolume:(CGFloat)volume { NSMutableArray *allAudioParams = [NSMutableArray array]; AVMutableAudioMixInputParameters *audioInputParams = [AVMutableAudioMixInputParameters audioMixInputParameters]; [audioInputParams setTrackID:_comTrack1.trackID]; _videoVolume = volume; [audioInputParams setVolume:_videoVolume atTime:kCMTimeZero]; [allAudioParams addObject:audioInputParams]; AVMutableAudioMixInputParameters *audioInputParams2 = [AVMutableAudioMixInputParameters audioMixInputParameters]; [audioInputParams2 setTrackID:_comTrack2.trackID]; [audioInputParams2 setVolume:_musicVolume atTime:kCMTimeZero]; [allAudioParams addObject:audioInputParams2]; AVMutableAudioMix *audioMix = [AVMutableAudioMix audioMix]; [audioMix setInputParameters:allAudioParams]; [_currentItem setAudioMix:audioMix]; }
导出实现
这里直接使用AVAssetExportSession来导出视频,与设置AVPlayerItem的audioMix属性相同,将audioMix设置给AVAssetExportSession实例即可导出混合的视频了
NSURL *outputFileUrl = [NSURL fileURLWithPath:outputPath]; AVAssetExportSession *_assetExport =[[AVAssetExportSession alloc]initWithAsset:self.composition presetName:AVAssetExportPreset1280x720]; _assetExport.outputFileType = AVFileTypeMPEG4; _assetExport.audioMix = _currentItem.audioMix; _assetExport.outputURL = outputFileUrl; _assetExport.shouldOptimizeForNetworkUse = YES; [_assetExport exportAsynchronouslyWithCompletionHandler:^{ // }];
最后贴上Demo地址 https://github.com/lucifron1994/VideoMixAudioDemo
以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持。