Share/Save

NOTE: Several of the non-alphanumeric characters do not copy and paste correctly.  If you have any problems (especially with the RunExternal actions), check that the quotes and dashes are correct.  Thanks to Steve for pointing that out in the comments.

Tools Used

• SageTV: records TV shows
• DVRMSToolbox: monitors recording folder and orchestrates conversion
  • SageTV Web Interface: provides metadata to DVRMSToolbox
  • ComSkip or ShowAnalyzer: finds commercials in TV show files for DTb
• mencoder (bundled with mplayer): removes commercials
• Handbrake: converts to h.264/aac in mp4 container
• AtomicParsley: writes metadata to mp4
• MoveOut: puts converted files in the “right” place

My Files

• “Convert to Zune/iPod/iPhone” Profile
• “Add to iTunes” Javascript (NOTE: I have tested this as a standalone script, but it is not incorporated into the workflow below)

Step 1: Installation and SageTV

Obviously, you need to have all of the above tools installed and SageTV configured and running and recording shows.  DVRMSToolbox (DTb) should be running and monitoring the SageTV recorded TV folder(s).

Step 2: Setup the DTb Profile

The problem I experienced with mencoder was due to the stdout mucking up DTb.  Turns out DTb has a RunExternalNoStatus action that ignores the output, thus obviating the problem.  Using this action for AtomicParsley sped up that app’s processing by tenfold (due to the huge amount of output of AP).  So, as a general rule, if you have having trouble getting an external app to run well (or at all) in DTb, try RunExternalNoStatus.

Now, for the meat of the process.  DTb orchestrates the various external applications.  The logical flow is:

1. Scan the TV show file for commercials.
2. Use mencoder to create a new TV show file without the commercials.
3. Use Handbrake to convert the file from MPEG to MP4 (h.264/AAC).
4. Use AtomicParsley to write metadata to the MP4 file (e.g., show, episode, air date, etc.).
5. Use DTb to rename the show to a friendly format (e.g., “Fringe – 2008-12-03 – Safe.mp4″).
6. Use MoveOut to move the shows to the correct folders (if necessary).

As implemented, here are the Actions and parameters:

1. FWThrottle: limit the number of files that are processed simultaneously
   1. UseDefaultSettings: <checked>
   2. ContextFile: InputFile
   3. Throttle: Checked
   4. ContinueOnFail: <unchecked>
2. CreateTimeout: sets a timeout – trust me, the first time mencoder hangs on “Dora the Explorer” and you don’t have “Lost” waiting for you in the morning like you were expecting, you’ll be glad.
   1. Timeout Minutes: 240
   2. ContinueOnFail: <unchecked>
3. ChangeContextValue: create a new context to store the location of the commercial file created by ComSkip/SA
   1. SourceContext: InputFile
   2. TargetContext: SageEDLFile
   3. CreateNewContext: <checked>
   4. NewContextName: SageEDLFile
   5. FindRegex: ^((.*\\)(.*)\.(.*))\z
   6. ReplaceRegex: $2$3.edl
   7. ContinueOnFail: <checked> (it’s OK to keep going if we fail for some reason – we just won’t get commercials cut out)
4. ChangeContextValue: update OutputFile context to allow for MP4 extension
   1. SourceContext: OutputFile
   2. TargetContext: OutputFile
   3. CreateNewContext: <unchecked>
   4. FindRegex: ^((.*\\)(.*)\.(.*))\z
   5. ReplaceRegex: $2$3.mp4
   6. ContinueOnFail: <unchecked>
5. ChangeContextValue: create a new context to specify location of the resulting commercial-free MPEG file
   1. SourceContext: OutputFile
   2. TargetContext: CutMpegFile
   3. CreateNewContext: <checked>
   4. NewContextName: CutMpegFile
   5. FindRegex: ^((.*\\)(.*)\.(.*))\z
   6. ReplaceRegex: $2$3.mpg
   7. ContinueOnFail: <unchecked>
6. FindCommercials: run ComSkip/SA to find commercials and write to EDL file
   1. UseDefaultSettings: <checked>
   2. ContextFile: InputFile
   3. ContinueOnFail: <checked> (again, worst case is we have commercials in the final MP4 file)
7. RunExternalNoStatus: run mencoder to cut out commercials
   1. ProcessName: <fullpath>\mencoder.exe
   2. ReadError: <unchecked>
   3. ArgumentsContext: “%InputFile%” -edl “%SageEDLFile%” -oac copy -ovc copy -of mpeg -o “%CutMpegFile%.mpg” -mpegopts muxrate=36000 -quiet
   4. ContinueOnFail: <unchecked>
8. RunExternalNoStatus: run Handbrake to convert to MP4
   1. ProcessName: <fullpath\handbrakeCLI.exe
   2. ReadError: <unchecked>
   3. ArgumentsContext: <experiment with the Handbrake GUI using a sample file to get the settings you want>
   4. ContinueOnFail: <unchecked>
9. MigrateMetadataToContext: workaround to get a nicely formatted “original air date”
   1. Metadata Pattern: %originalAirDate%
   2. TargetContext: OutputFile
   3. CreateNewContext: <checked>
   4. NewContextName: DateAndEpisode
   5. FindRegex: ((\d+-)+\d+)
   6. ReplaceRegex: $1
   7. ContinueOnFail: <unchecked>
10. RunExternalNoStatus: run AtomicParsley to add basic metadata to MP4
    1. Processname: <fullpath>\AtomicParsley.exe
    2. ReadError: <unchecked>
    3. ArgumentsContext: “%OutputFile%” –genre “TV Shows” –stik “TV Show” –TVNetwork %channelName% –TVShowName “%Title%” –TVEpisode “%episode%” –description “%description%” –category=”%category%”
    4. ContinueOnFail: <checked> (no need to crash just for the metadata)
11. AdvancedFileRename: rename the output file to something meaningful
    1. ContextFile: OutputFile
    2. NamePattern: %Title% – %originalAirDate% – %episode%
    3. WhenFileExists: AppendGuid
12. FileOperation: delete temporary MPEG file
    1. ContextFile: CutMpegFile
    2. DestinationPath: <blank>
    3. WhatOperation: Delete
13. FileOperation: move MP4 to the output directory
    1. ContextFile: OutputFile
    2. DestinationPath: <destination directory>
    3. WhatOperation: Move

Step 3: Move the files

I use MoveOut to then put some files in the kids’ directory, some in the wife’s Zune directory, and some in my directory.  You could do this all with DTb, but I find it easier to use a separate app.

What’s Next?

Zune users are all set – the ZMP automatically monitors folders, so the next time you sync your Zune, it will pick up the new MP4 files.  iPod/iPhone users, however, have to manually import the folder prior to each sync.  There are two options for getting around this.  First, you can add an extra “RunExternal” action that runs a Javascript to import the files into iTunes (available in the linked download, courtesy of Coyote).  Second, you can try to get the iTunes Library Updater to work, although there are mixed results with iTunes 7+.  I don’t have iTunes installed on my SageTV server, so this isn’t really an issue for me.

jelwood is working on an alternate approach using Quicktime chapters instead of actually cutting out the commercials.  Because not every medium I am watching these on supports Quicktime chapters (including SageTV and the Zune), I am sticking with the mencoder route.  However, this is a better option if storage space is not an issue or you have problems with ComSkip/SA getting the commercials “not quite right”.

Also, jelwood has created a patched version of AtomicParsley that allows you to set the sort order to the original air date.  I include the air date in my titles out of preference, so this is not necessary for me, but incorporating this is trivial.

Anything else?

EDIT (2009-03-18): DTb 1.2.1.6 added the MigrateMetadataToContext, which makes it easier to format the original air date to a friendlier format.

Share/Save

Andy has posted yet another great tool on his site: Commercial Viewer.  This tool loads a video file and its associated EDL commercial file and lets you modify the values to fine-tune the commercial segments.  This isn’t a tool for every show that gets recorded – I use this for shows I want to archive indefinitely.

In the screenshot above, you can see the app.  Once you select a video, it also loads the associated EDL file (of course, you have to run SA or ComSkip on the video file first).  You can preview the video, although you can’t jump to an arbitrary spot (i.e., there is no slider) and you can jump to an arbitrary spot by clicking on the green/red slider (updated per Andy’s comment below – the perils of 5a posting).  You can select a commercial segment using the list on the right.  The two text boxes display the start and end times of the commercial segment, and you can jump to either of those spots.

Get it here.

Share/Save

NOTE: mencoder is not cutting commercials properly on my system.  Everything else is working fine.  I’ll post an update when I get it working.

Here is how I’m converting SageTV recordings to iPhone-compatible (and Touch/iPod-compatible) MP4 videos.  This will also work for the Zune!

At a high level, the steps (and applications used) are:

1. DVRMSToolbox: monitor the recordings directories, identify which shows to process, grab the metadata that is later written to the MP4, and control program flow
2. comskip or ShowAnalyzer: Scan for commercials
3. mencoder: Cut commercials
4. ffmpeg: Convert to MP4 (note: I had problems with the latest Windows build of ffmpeg, but the one from babgvant.com worked fine)
   1. Option 2 is to use mencoder to cut out commercials and convert to MP4 in single pass, and then use NicMP4Box to fix the MP4 container to make it iPod-compatible
      • Apparently mencoder’s current implementation (2008-09-17) of lavf correctly handles the MP4 container, but I haven’t verified this.  If true, you can skip this step.
      • Also (apparently), the latest SVN of AtomicParsley can fix this, but you have to build it yourself.
        
5. AtomicParsley: Write metadata to identify as TV Shows (plus any other info we have)

I am assuming basic familiarity with DTb, so I’ll talk about how to setup the Profile to process a Sage recording, but I am not addressing installing DTb or setting up the processing condition.

More after the break…

DTb: Setup the Profiles

Note you will need to create one profile for SD and one for HD.  The only difference will be profile indicated in the the call to mencoder/ffmpeg (step 6).

1. Throttle to prevent unlimited workers from running
2. Set a timeout (trust me – the first time mencoder hangs and you don’t have “Lost” waiting for you in the morning like you were expecting, you’ll be glad)
3. Update OutputFile context to allow for MP4 extension:
   FIND IN OutputFile: ^((.*\)(.*).(.*))z
   REPLACE IN OutputFile: $2$3.mp4
4. Create a new context to specify the Sage commercial file location:
   FIND IN InputFile: ^((.*\)(.*).(.*))z
   REPLACE IN SageEDLFile: $2$3.edl
5. Find commercials
6. Option 1 – ffmpeg
   Note: this process is tried and true, but you suffer a performance hit since mencoder has to rewrite the entire file to cut the commercials. 
   1. Cut commercials with mencoder
      mencoder “%InputFile%” -edl “%SageEDLFile%” -oac copy -ovc copy -of mpeg -o “%OutputFile%.mpg”
   2. Convert with ffmpeg
      Profile: X264 with AAC
7. Option 2 – mencoder
   Note: mencoder does not have the best implementation of the mp4 container, and I have had problems getting the conversion to work properly.  However, if you can get it to work, the overall process is faster) 
   1. Cut and convert with mencoder
      mencoder “%InputFile%” -edl “%SageEDLFile%” -profile x264IPod-SD -o “%OutputFile%.tmp.mp4″
   2. Remux MP4 with NicMP4Box
      NicMP4Box -add “%OutputFile%.TMP.mp4″ “%OutputFile%”
8. Add basic metadata to MP4:
   AtomicParsley “%OutputFile%” –genre “TV Shows” –stik “TV Show” –TVNetwork %channelName% –TVShowName “%Title%” –TVEpisode “%episode%” –description “%description%” –category=”%category%”
9. Rename the output file to something meaningful:
   CONTEXT: OutputFile
   PATTERN: %Title% – %originalAirDate% – %episode%

Setup the Profiles

I use two profiles, one for HD sources and one for SD.  I resize to 640×480 or 640×360, since that’s the max resolution the iPhone can handle (Zune can handle 720×480).

A couple of other notes:

• You must use H.264 or MPEG-4 with AAC audio – the iPhone supports nothing else.
• You can mess with the bitrates (bitrate=x), but keep in mind the bitrate maximums (vbv_maxrate=x) that the iPhone supports (currently 1500).
• The maximum IDC is currently 30 (level_idc=30)
• Cabac is not supported (nocabac)

Option 1: ffmpeg

Here’s the ffmpeg profile that’s working for me right now (note the “-ac 2″ parameter for the audio encoding – this was the only way I could get the sound to play on the iPhone):

<profile>
<name>MPEG-4 with AAC HD</name>
<description>MPEG-4 with AAC HD (iPhone-compatible Archive)</description>
<extension>mp4</extension>
<video>-vcodec mpeg4 -b 1200kb -mbd 2 -flags +4mv+trell -aic 2 -cmp 2 -subcmp 2</video>
<audio>-acodec libfaac -ac 2 -ab 128000</audio>
</profile>

Here are some other command lines that I’ve found, mostly from here, in case you want to tinker:

• ffmpeg -y -benchmark -threads 2 -i “input.dvr-ms” -async 1 -deinterlace -s qvga -vcodec libx264 -b 384000 -bufsize 384 -coder 0 -level 13 -acodec libfaac “output.mp4″
• 4:3 and 16:9:
  • ffmpeg -y -benchmark -threads 2 -i “%CutDvrmsFile%” -async 1 -deinterlace -croptop 0 -cropbottom 4 -cropleft 8 -cropright 6 -s 704:480 -vcodec libx264 -b 832k -maxrate 960k -bufsize 384k -coder 1 -level 13  -flags +loop -cmp +chroma -partitions +parti4x4+parti8x8+partp4x4+partp8x8+partb8x8 -flags2 +wpred+mixed_refs+8x8dct+brdo -me umh -trellis 2 -bidir_refine 1 -refs 3 -b_strategy 1 -directpred 2 -subq 5 -me_range 16 -g 250 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -aspect 4:3 -acodec libfaac -ab 128k “%OutputFile%”
  • ffmpeg -y -benchmark -threads 2 -i “%CutDvrmsFile%” -async 1 -deinterlace -croptop 56 -cropbottom 60 -cropleft 8 -cropright 6 -s 704:368 -vcodec libx264 -b 832k -maxrate 960k -bufsize 384k -coder 1 -level 13  -flags +loop -cmp +chroma -partitions +parti4x4+parti8x8+partp4x4+partp8x8+partb8x8 -flags2 +wpred+mixed_refs+8x8dct+brdo -me umh -trellis 2 -bidir_refine 1 -refs 3 -b_strategy 1 -directpred 2 -subq 5 -me_range 16 -g 250 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -aspect 16:9 -acodec libfaac -ab 128k “%OutputFile%”
• ffmpeg -i input -acodec libfaac -ab 128kb -vcodec mpeg4 -b 1200kb -mbd 2 -flags +4mv+trell -aic 2 -cmp 2 -subcmp 2 -s 640×360 -title X output.mp4
  (from http://ffmpeg.mplayerhq.hu/faq.html#SEC25)

Option 2: mencoder

[x264IPod-SD]
profile-desc=”H.264/AAC MP4 SD for iPhone –After this, use NicMP4Box to remux–”
lavdopts=threads=2
vf=pp=md,scale=640:-10,harddup
ovc=x264=yes
x264encopts=turbo=1:bitrate=768:vbv_maxrate=1500:vbv_bufsize=244:nocabac:level_idc=30:threads=2
oac=faac=yes
faacopts=br=128:object=2:mpeg=4:raw=yes
of=lavf=yes
lavfopts=format=mp4
#DtbExtension=mp4
#dtbhres=”-vf pp=md,scale={0}:-10,harddup”

[x264IPod-HD]
profile-desc=”H.264/AAC MP4 SD for iPhone –After this, use NicMP4Box to remux–”
lavdopts=threads=2
vf=pp=md,scale=640:-10,harddup
ovc=x264=yes
x264encopts=turbo=1:bitrate=1200:vbv_maxrate=1500:vbv_bufsize=244:nocabac:level_idc=30:threads=2
oac=faac=yes
faacopts=br=128:object=2:mpeg=4:raw=yes
of=lavf=yes
lavfopts=format=mp4
#DtbExtension=mp4
#dtbhres=”-vf pp=md,scale={0}:-10,harddup”

Resources:

• http://lists.mplayerhq.hu/pipermail/mencoder-users/2007-December/007692.html
• http://rob.opendot.cl/index.php/useful-stuff/ipod-video-guide/
• Getting mencoder to cut commercials with muxing errors:
  • http://forum.videohelp.com/topic346473.html
  • http://readlist.com/lists/mplayerhq.hu/mencoder-users/0/3146.html
  • Increase muxrate to 36000
  • -noskip option

Related posts:

• http://onlydarksets.com/2008/09/16/transcoding-in-sage/
• http://onlydarksets.com/2008/07/06/mp4-tagging/
• http://onlydarksets.com/2008/06/13/convert-for-iphone-using-mencoderffmpeg/

Share/Save

A couple more resources:

• mencoder build that can cut/merge MPEG files without recoding
• ffmpeg Windows binaries
• Atomic Parsley (for writing MP4 metadata)
• Atomic Parsely Windows GUI (just found this – haven’t tested it)
• Coyote’s iPod guide for DTb

Share/Save

The switch is “/VideoComplexity 0″, and it reduces the computing time to process the video.  I never noticed a difference between setting it to 3 or dropping it to 0, but YMMV.

“C:\Program Files\Microsoft Expression\Encoder 1.0\Encoder.exe” /Source “e:\HD_SOURCE.dvr-ms” /Target “E:\HD_TARGET.wmv” /VideoComplexity 0 /VideoProfile “HD Archive (CBR)” /Log On /Time

“C:\Program Files\Microsoft Expression\Encoder 1.0\Encoder.exe” /Source “e:\SD_SOURCE.dvr-ms” /Target “E:\SD_TARGET.wmv” /VideoComplexity 0 /VideoProfile “SD Archive (CBR)” /Log On /Time

I setup two different profiles (SD and HD), and had each bring the resolution down to 720×480 for SD or 720×404 for HD.  Of course, this was all automated with commericials removed by DTb.

Share/Save

Well, gskinner to the rescue with a web app for this (Flash required).  There’s also a desktop version (Adobe Air required).

There is also a good guide from Jan Goyvaerts.

Share/Save

This Action will write certain meta data to the CutDvrmsFile Context. By default, it will create two new meta data fields (and default them to the following values):

• WM/Toolname = DVRMSToolbox
• WM/ToolVersion = <currentversion> (e.g., 1.2.1.0)

You can also do the following:

• FixSlider – not sure why you would do this, but it sets the WM/WMRVEndTime and WM/MediaOriginalRunTime fields.
• AddTooledToDescription – prepends the MetaMarker value to the WM/SubTitleDescription field. You can then check this value in a processing condition to see if the file has run through a particular Profile.
• IncrementScheduleItem – not sure why you would do this, but it increments WM/WMRVScheduleItemID.

Using Toub’s DVR-MS meta data editor, you can see the following meta data fields and values have been added:

A few important notes:

1. AdjustMetaData only writes to the CutDVRMSFile Context. So, if you want to edit the input file (for example), you need to first copy the InputFile Context to the CutDVRMSFile Context. You can set the CutDVRMSFile value by:
   1. Add a ChangeContextValue Action
   2. SourceContext = <Context for which you want to update the meta data>
   3. TargetContext = CutDVRMSFile
   4. FindRegex = (.*)
   5. ReplaceRegex = $1
2. This should be the last Action in your Profile. Otherwise, if there is a failure, the file will be marked as successfully processed when, in fact, it was not. Also, due to #1, if you set it too early you could muck up another Action that legitimately needs to use the CutDVRMSFile Context (e.g., cutting commercials from DVR-MS).
3. Regenerate MetaTags.bin. MetaTags.bin is a list of all meta data that can be evaluated by a Processing Condition. The “stock” MetaTags.bin does not include WM/Toolname. If you want to evaluate WM/Toolname, you need to first run a Profile that sets WM/Toolname on a DVR-MS file, and then run Generate MetaTags.bin from the Processing Condition dialog box (and select the DVR-MS file you just modified).

Share/Save

FindRegex: ^((.*\\)(.*)\.(.*))\z

ReplaceRegex:

• $1 = E:\Recorded TV\subdir\another one\Dirty Sexy Money_WJLA-DT_10_10_2007_22_00_01.dvr-ms
• $2 = E:\Recorded TV\subdir\another one\
• $3 = Dirty Sexy Money_WJLA-DT_10_10_2007_22_00_01
• $4 = dvr-ms

So, if you want to change the extension, the ReplaceRegex value would be:

$2$3.wmv

If you want to change the path, the ReplaceRegex would be:

C:\New Path\$3.$4

I don’t suggest using this to change the filename (use the AdvancedFileRename Action instead). However, if you insist, the ReplaceRegex would be:

$2$3 (FIXED).$4

Share/Save

Has anyone found a single set of video conversion settings that will allow proper aspect ratios on both the Zune and through Vista Media Center (or an extender)?

DVR-MS SD shows are natively 720×480.  If I maintain the aspect ratio when converting to WMV, the file plays fine on the Zune, but it is letterboxed.  If I lower the resolution to 640×480 during conversion, it looks perfect on the Zune, but the AR is obviously wrong when I play it on VMC or an extender.

I am looking for a conversion setting that will allow the video to display with the proper aspect ratio, regardless of where it is played.  Alternatively, is there an option on the Zune to “stretch” the video to fit the screen during playback?

Share/Save