Share/Save

Turns out you can access most of Windows Media Player via WSH, so that made it pretty easy to grab all of the metadata once everything was loaded into WMP – you just have to know where to look.  The XML layout for the Sage Webserver is pretty well documented, although the source code to Andy’s excellent MigrateMCtoSage utility made it much easier to figure out the bare minimums.  The hardest part was actually finding the XML writer for WSH.

Note that this should also work for AVIs, which is the other main container format supported by WMP.  However, AVI tagging is pretty abysmal in WMP, mostly because it doesn’t write the tags back to the file.  I have found a couple of apps to do this, but nothing I would wholeheartedly recommend.

Backup you wiz.bin file before proceeding – you need to shut down the Sage service to do this properly.

1. Download and unzip CreateVideoXML.vbs
   Optional: Open in a text editor and edit the output and temp files
2. Import your videos into WMP and clean up the metadata
3. Run CreateVideoXML.vbs
4. Open Sage Webserver and import the resulting XML file (by default: C:\Playlist.xml)
   1. Check the Import TV files box
   2. Check the Overwrite Existing Show Information box
   3. Check the Overwrite Existing Sage TV File Information box
   4. Check the Use imported Airing data to determine startTime option (THIS IS VERY IMPORTANT)
   5. Uncheck everything else

After import, your videos will show up under “My TV”.

Share/Save

NOTE: Several of the non-alphanumeric characters do not copy and paste correctly.  If you have any problems (especially with the RunExternal actions), check that the quotes and dashes are correct.  Thanks to Steve for pointing that out in the comments.

Tools Used

• SageTV: records TV shows
• DVRMSToolbox: monitors recording folder and orchestrates conversion
  • SageTV Web Interface: provides metadata to DVRMSToolbox
  • ComSkip or ShowAnalyzer: finds commercials in TV show files for DTb
• mencoder (bundled with mplayer): removes commercials
• Handbrake: converts to h.264/aac in mp4 container
• AtomicParsley: writes metadata to mp4
• MoveOut: puts converted files in the “right” place

My Files

• “Convert to Zune/iPod/iPhone” Profile
• “Add to iTunes” Javascript (NOTE: I have tested this as a standalone script, but it is not incorporated into the workflow below)

Step 1: Installation and SageTV

Obviously, you need to have all of the above tools installed and SageTV configured and running and recording shows.  DVRMSToolbox (DTb) should be running and monitoring the SageTV recorded TV folder(s).

Step 2: Setup the DTb Profile

The problem I experienced with mencoder was due to the stdout mucking up DTb.  Turns out DTb has a RunExternalNoStatus action that ignores the output, thus obviating the problem.  Using this action for AtomicParsley sped up that app’s processing by tenfold (due to the huge amount of output of AP).  So, as a general rule, if you have having trouble getting an external app to run well (or at all) in DTb, try RunExternalNoStatus.

Now, for the meat of the process.  DTb orchestrates the various external applications.  The logical flow is:

1. Scan the TV show file for commercials.
2. Use mencoder to create a new TV show file without the commercials.
3. Use Handbrake to convert the file from MPEG to MP4 (h.264/AAC).
4. Use AtomicParsley to write metadata to the MP4 file (e.g., show, episode, air date, etc.).
5. Use DTb to rename the show to a friendly format (e.g., “Fringe – 2008-12-03 – Safe.mp4″).
6. Use MoveOut to move the shows to the correct folders (if necessary).

As implemented, here are the Actions and parameters:

1. FWThrottle: limit the number of files that are processed simultaneously
   1. UseDefaultSettings: <checked>
   2. ContextFile: InputFile
   3. Throttle: Checked
   4. ContinueOnFail: <unchecked>
2. CreateTimeout: sets a timeout – trust me, the first time mencoder hangs on “Dora the Explorer” and you don’t have “Lost” waiting for you in the morning like you were expecting, you’ll be glad.
   1. Timeout Minutes: 240
   2. ContinueOnFail: <unchecked>
3. ChangeContextValue: create a new context to store the location of the commercial file created by ComSkip/SA
   1. SourceContext: InputFile
   2. TargetContext: SageEDLFile
   3. CreateNewContext: <checked>
   4. NewContextName: SageEDLFile
   5. FindRegex: ^((.*\\)(.*)\.(.*))\z
   6. ReplaceRegex: $2$3.edl
   7. ContinueOnFail: <checked> (it’s OK to keep going if we fail for some reason – we just won’t get commercials cut out)
4. ChangeContextValue: update OutputFile context to allow for MP4 extension
   1. SourceContext: OutputFile
   2. TargetContext: OutputFile
   3. CreateNewContext: <unchecked>
   4. FindRegex: ^((.*\\)(.*)\.(.*))\z
   5. ReplaceRegex: $2$3.mp4
   6. ContinueOnFail: <unchecked>
5. ChangeContextValue: create a new context to specify location of the resulting commercial-free MPEG file
   1. SourceContext: OutputFile
   2. TargetContext: CutMpegFile
   3. CreateNewContext: <checked>
   4. NewContextName: CutMpegFile
   5. FindRegex: ^((.*\\)(.*)\.(.*))\z
   6. ReplaceRegex: $2$3.mpg
   7. ContinueOnFail: <unchecked>
6. FindCommercials: run ComSkip/SA to find commercials and write to EDL file
   1. UseDefaultSettings: <checked>
   2. ContextFile: InputFile
   3. ContinueOnFail: <checked> (again, worst case is we have commercials in the final MP4 file)
7. RunExternalNoStatus: run mencoder to cut out commercials
   1. ProcessName: <fullpath>\mencoder.exe
   2. ReadError: <unchecked>
   3. ArgumentsContext: “%InputFile%” -edl “%SageEDLFile%” -oac copy -ovc copy -of mpeg -o “%CutMpegFile%.mpg” -mpegopts muxrate=36000 -quiet
   4. ContinueOnFail: <unchecked>
8. RunExternalNoStatus: run Handbrake to convert to MP4
   1. ProcessName: <fullpath\handbrakeCLI.exe
   2. ReadError: <unchecked>
   3. ArgumentsContext: <experiment with the Handbrake GUI using a sample file to get the settings you want>
   4. ContinueOnFail: <unchecked>
9. MigrateMetadataToContext: workaround to get a nicely formatted “original air date”
   1. Metadata Pattern: %originalAirDate%
   2. TargetContext: OutputFile
   3. CreateNewContext: <checked>
   4. NewContextName: DateAndEpisode
   5. FindRegex: ((\d+-)+\d+)
   6. ReplaceRegex: $1
   7. ContinueOnFail: <unchecked>
10. RunExternalNoStatus: run AtomicParsley to add basic metadata to MP4
    1. Processname: <fullpath>\AtomicParsley.exe
    2. ReadError: <unchecked>
    3. ArgumentsContext: “%OutputFile%” –genre “TV Shows” –stik “TV Show” –TVNetwork %channelName% –TVShowName “%Title%” –TVEpisode “%episode%” –description “%description%” –category=”%category%”
    4. ContinueOnFail: <checked> (no need to crash just for the metadata)
11. AdvancedFileRename: rename the output file to something meaningful
    1. ContextFile: OutputFile
    2. NamePattern: %Title% – %originalAirDate% – %episode%
    3. WhenFileExists: AppendGuid
12. FileOperation: delete temporary MPEG file
    1. ContextFile: CutMpegFile
    2. DestinationPath: <blank>
    3. WhatOperation: Delete
13. FileOperation: move MP4 to the output directory
    1. ContextFile: OutputFile
    2. DestinationPath: <destination directory>
    3. WhatOperation: Move

Step 3: Move the files

I use MoveOut to then put some files in the kids’ directory, some in the wife’s Zune directory, and some in my directory.  You could do this all with DTb, but I find it easier to use a separate app.

What’s Next?

Zune users are all set – the ZMP automatically monitors folders, so the next time you sync your Zune, it will pick up the new MP4 files.  iPod/iPhone users, however, have to manually import the folder prior to each sync.  There are two options for getting around this.  First, you can add an extra “RunExternal” action that runs a Javascript to import the files into iTunes (available in the linked download, courtesy of Coyote).  Second, you can try to get the iTunes Library Updater to work, although there are mixed results with iTunes 7+.  I don’t have iTunes installed on my SageTV server, so this isn’t really an issue for me.

jelwood is working on an alternate approach using Quicktime chapters instead of actually cutting out the commercials.  Because not every medium I am watching these on supports Quicktime chapters (including SageTV and the Zune), I am sticking with the mencoder route.  However, this is a better option if storage space is not an issue or you have problems with ComSkip/SA getting the commercials “not quite right”.

Also, jelwood has created a patched version of AtomicParsley that allows you to set the sort order to the original air date.  I include the air date in my titles out of preference, so this is not necessary for me, but incorporating this is trivial.

Anything else?

EDIT (2009-03-18): DTb 1.2.1.6 added the MigrateMetadataToContext, which makes it easier to format the original air date to a friendlier format.

Share/Save

Andy has posted yet another great tool on his site: Commercial Viewer.  This tool loads a video file and its associated EDL commercial file and lets you modify the values to fine-tune the commercial segments.  This isn’t a tool for every show that gets recorded – I use this for shows I want to archive indefinitely.

In the screenshot above, you can see the app.  Once you select a video, it also loads the associated EDL file (of course, you have to run SA or ComSkip on the video file first).  You can preview the video, although you can’t jump to an arbitrary spot (i.e., there is no slider) and you can jump to an arbitrary spot by clicking on the green/red slider (updated per Andy’s comment below – the perils of 5a posting).  You can select a commercial segment using the list on the right.  The two text boxes display the start and end times of the commercial segment, and you can jump to either of those spots.

Get it here.

Share/Save

NOTE: mencoder is not cutting commercials properly on my system.  Everything else is working fine.  I’ll post an update when I get it working.

Here is how I’m converting SageTV recordings to iPhone-compatible (and Touch/iPod-compatible) MP4 videos.  This will also work for the Zune!

At a high level, the steps (and applications used) are:

1. DVRMSToolbox: monitor the recordings directories, identify which shows to process, grab the metadata that is later written to the MP4, and control program flow
2. comskip or ShowAnalyzer: Scan for commercials
3. mencoder: Cut commercials
4. ffmpeg: Convert to MP4 (note: I had problems with the latest Windows build of ffmpeg, but the one from babgvant.com worked fine)
   1. Option 2 is to use mencoder to cut out commercials and convert to MP4 in single pass, and then use NicMP4Box to fix the MP4 container to make it iPod-compatible
      • Apparently mencoder’s current implementation (2008-09-17) of lavf correctly handles the MP4 container, but I haven’t verified this.  If true, you can skip this step.
      • Also (apparently), the latest SVN of AtomicParsley can fix this, but you have to build it yourself.
        
5. AtomicParsley: Write metadata to identify as TV Shows (plus any other info we have)

I am assuming basic familiarity with DTb, so I’ll talk about how to setup the Profile to process a Sage recording, but I am not addressing installing DTb or setting up the processing condition.

More after the break…

DTb: Setup the Profiles

Note you will need to create one profile for SD and one for HD.  The only difference will be profile indicated in the the call to mencoder/ffmpeg (step 6).

1. Throttle to prevent unlimited workers from running
2. Set a timeout (trust me – the first time mencoder hangs and you don’t have “Lost” waiting for you in the morning like you were expecting, you’ll be glad)
3. Update OutputFile context to allow for MP4 extension:
   FIND IN OutputFile: ^((.*\)(.*).(.*))z
   REPLACE IN OutputFile: $2$3.mp4
4. Create a new context to specify the Sage commercial file location:
   FIND IN InputFile: ^((.*\)(.*).(.*))z
   REPLACE IN SageEDLFile: $2$3.edl
5. Find commercials
6. Option 1 – ffmpeg
   Note: this process is tried and true, but you suffer a performance hit since mencoder has to rewrite the entire file to cut the commercials. 
   1. Cut commercials with mencoder
      mencoder “%InputFile%” -edl “%SageEDLFile%” -oac copy -ovc copy -of mpeg -o “%OutputFile%.mpg”
   2. Convert with ffmpeg
      Profile: X264 with AAC
7. Option 2 – mencoder
   Note: mencoder does not have the best implementation of the mp4 container, and I have had problems getting the conversion to work properly.  However, if you can get it to work, the overall process is faster) 
   1. Cut and convert with mencoder
      mencoder “%InputFile%” -edl “%SageEDLFile%” -profile x264IPod-SD -o “%OutputFile%.tmp.mp4″
   2. Remux MP4 with NicMP4Box
      NicMP4Box -add “%OutputFile%.TMP.mp4″ “%OutputFile%”
8. Add basic metadata to MP4:
   AtomicParsley “%OutputFile%” –genre “TV Shows” –stik “TV Show” –TVNetwork %channelName% –TVShowName “%Title%” –TVEpisode “%episode%” –description “%description%” –category=”%category%”
9. Rename the output file to something meaningful:
   CONTEXT: OutputFile
   PATTERN: %Title% – %originalAirDate% – %episode%

Setup the Profiles

I use two profiles, one for HD sources and one for SD.  I resize to 640×480 or 640×360, since that’s the max resolution the iPhone can handle (Zune can handle 720×480).

A couple of other notes:

• You must use H.264 or MPEG-4 with AAC audio – the iPhone supports nothing else.
• You can mess with the bitrates (bitrate=x), but keep in mind the bitrate maximums (vbv_maxrate=x) that the iPhone supports (currently 1500).
• The maximum IDC is currently 30 (level_idc=30)
• Cabac is not supported (nocabac)

Option 1: ffmpeg

Here’s the ffmpeg profile that’s working for me right now (note the “-ac 2″ parameter for the audio encoding – this was the only way I could get the sound to play on the iPhone):

<profile>
<name>MPEG-4 with AAC HD</name>
<description>MPEG-4 with AAC HD (iPhone-compatible Archive)</description>
<extension>mp4</extension>
<video>-vcodec mpeg4 -b 1200kb -mbd 2 -flags +4mv+trell -aic 2 -cmp 2 -subcmp 2</video>
<audio>-acodec libfaac -ac 2 -ab 128000</audio>
</profile>

Here are some other command lines that I’ve found, mostly from here, in case you want to tinker:

• ffmpeg -y -benchmark -threads 2 -i “input.dvr-ms” -async 1 -deinterlace -s qvga -vcodec libx264 -b 384000 -bufsize 384 -coder 0 -level 13 -acodec libfaac “output.mp4″
• 4:3 and 16:9:
  • ffmpeg -y -benchmark -threads 2 -i “%CutDvrmsFile%” -async 1 -deinterlace -croptop 0 -cropbottom 4 -cropleft 8 -cropright 6 -s 704:480 -vcodec libx264 -b 832k -maxrate 960k -bufsize 384k -coder 1 -level 13  -flags +loop -cmp +chroma -partitions +parti4x4+parti8x8+partp4x4+partp8x8+partb8x8 -flags2 +wpred+mixed_refs+8x8dct+brdo -me umh -trellis 2 -bidir_refine 1 -refs 3 -b_strategy 1 -directpred 2 -subq 5 -me_range 16 -g 250 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -aspect 4:3 -acodec libfaac -ab 128k “%OutputFile%”
  • ffmpeg -y -benchmark -threads 2 -i “%CutDvrmsFile%” -async 1 -deinterlace -croptop 56 -cropbottom 60 -cropleft 8 -cropright 6 -s 704:368 -vcodec libx264 -b 832k -maxrate 960k -bufsize 384k -coder 1 -level 13  -flags +loop -cmp +chroma -partitions +parti4x4+parti8x8+partp4x4+partp8x8+partb8x8 -flags2 +wpred+mixed_refs+8x8dct+brdo -me umh -trellis 2 -bidir_refine 1 -refs 3 -b_strategy 1 -directpred 2 -subq 5 -me_range 16 -g 250 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -aspect 16:9 -acodec libfaac -ab 128k “%OutputFile%”
• ffmpeg -i input -acodec libfaac -ab 128kb -vcodec mpeg4 -b 1200kb -mbd 2 -flags +4mv+trell -aic 2 -cmp 2 -subcmp 2 -s 640×360 -title X output.mp4
  (from http://ffmpeg.mplayerhq.hu/faq.html#SEC25)

Option 2: mencoder

[x264IPod-SD]
profile-desc=”H.264/AAC MP4 SD for iPhone –After this, use NicMP4Box to remux–”
lavdopts=threads=2
vf=pp=md,scale=640:-10,harddup
ovc=x264=yes
x264encopts=turbo=1:bitrate=768:vbv_maxrate=1500:vbv_bufsize=244:nocabac:level_idc=30:threads=2
oac=faac=yes
faacopts=br=128:object=2:mpeg=4:raw=yes
of=lavf=yes
lavfopts=format=mp4
#DtbExtension=mp4
#dtbhres=”-vf pp=md,scale={0}:-10,harddup”

[x264IPod-HD]
profile-desc=”H.264/AAC MP4 SD for iPhone –After this, use NicMP4Box to remux–”
lavdopts=threads=2
vf=pp=md,scale=640:-10,harddup
ovc=x264=yes
x264encopts=turbo=1:bitrate=1200:vbv_maxrate=1500:vbv_bufsize=244:nocabac:level_idc=30:threads=2
oac=faac=yes
faacopts=br=128:object=2:mpeg=4:raw=yes
of=lavf=yes
lavfopts=format=mp4
#DtbExtension=mp4
#dtbhres=”-vf pp=md,scale={0}:-10,harddup”

Resources:

• http://lists.mplayerhq.hu/pipermail/mencoder-users/2007-December/007692.html
• http://rob.opendot.cl/index.php/useful-stuff/ipod-video-guide/
• Getting mencoder to cut commercials with muxing errors:
  • http://forum.videohelp.com/topic346473.html
  • http://readlist.com/lists/mplayerhq.hu/mencoder-users/0/3146.html
  • Increase muxrate to 36000
  • -noskip option

Related posts:

• http://onlydarksets.com/2008/09/16/transcoding-in-sage/
• http://onlydarksets.com/2008/07/06/mp4-tagging/
• http://onlydarksets.com/2008/06/13/convert-for-iphone-using-mencoderffmpeg/

Share/Save

A couple more resources:

• mencoder build that can cut/merge MPEG files without recoding
• ffmpeg Windows binaries
• Atomic Parsley (for writing MP4 metadata)
• Atomic Parsely Windows GUI (just found this – haven’t tested it)
• Coyote’s iPod guide for DTb

Share/Save

Following Andy’s lead, I just upgraded my license from Sage 4 to 6 and ordered an extender.

Share/Save

Share/Save